peepmatic: Introduce the peepmatic-automata crate

The `peepmatic-automata` crate builds and queries finite-state transducer
automata.

A transducer is a type of automata that has not only an input that it
accepts or rejects, but also an output. While regular automata check whether
an input string is in the set that the automata accepts, a transducer maps
the input strings to values. A regular automata is sort of a compressed,
immutable set, and a transducer is sort of a compressed, immutable key-value
dictionary. A [trie] compresses a set of strings or map from a string to a
value by sharing prefixes of the input string. Automata and transducers can
compress even better: they can share both prefixes and suffixes. [*Index
1,600,000,000 Keys with Automata and Rust* by Andrew Gallant (aka
burntsushi)][burntsushi-blog-post] is a top-notch introduction.

If you're looking for a general-purpose transducers crate in Rust you're
probably looking for [the `fst` crate][fst-crate]. While this implementation
is fully generic and has no dependencies, its feature set is specific to
`peepmatic`'s needs:

* We need to associate extra data with each state: the match operation to
  evaluate next.

* We can't provide the full input string up front, so this crate must
  support incremental lookups. This is because the peephole optimizer is
  computing the input string incrementally and dynamically: it looks at the
  current state's match operation, evaluates it, and then uses the result as
  the next character of the input string.

* We also support incremental insertion and output when building the
  transducer. This is necessary because we don't want to emit output values
  that bind a match on an optimization's left-hand side's pattern (for
  example) until after we've succeeded in matching it, which might not
  happen until we've reached the n^th state.

* We need to support generic output values. The `fst` crate only supports
  `u64` outputs, while we need to build up an optimization's right-hand side
  instructions.

This implementation is based on [*Direct Construction of Minimal Acyclic
Subsequential Transducers* by Mihov and Maurel][paper]. That means that keys
must be inserted in lexicographic order during construction.

[trie]: https://en.wikipedia.org/wiki/Trie
[burntsushi-blog-post]: https://blog.burntsushi.net/transducers/#ordered-maps
[fst-crate]: https://crates.io/crates/fst
[paper]: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.24.3698&rep=rep1&type=pdf
This commit is contained in:
Nick Fitzgerald
2020-05-01 15:30:37 -07:00
parent 0592b5a995
commit c82326a1ae
5 changed files with 1603 additions and 0 deletions

View File

@@ -0,0 +1,18 @@
[package]
name = "peepmatic-automata"
version = "0.1.0"
authors = ["Nick Fitzgerald <fitzgen@gmail.com>"]
edition = "2018"
[package.metadata.docs.rs]
all-features = true
[dependencies]
serde = { version = "1.0.106", optional = true }
[features]
# Enable support for generating GraphViz Dot files that can be used to visually
# render an automaton.
#
# https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29
dot = []

View File

@@ -0,0 +1,273 @@
//! Helpers for generating [GraphViz
//! Dot](https://graphviz.gitlab.io/_pages/pdf/dotguide.pdf) files to visually
//! render automata.
//!
//! **This module only exists when the `"dot"` cargo feature is enabled.**
use crate::{Automaton, Output, State};
use std::fmt::{Debug, Display};
use std::fs;
use std::hash::Hash;
use std::io::{self, Write};
use std::path::Path;
/// Format the user-provided bits of an `Automaton` for Graphviz Dot output.
///
/// There are two provided implementations of `DotFmt`:
///
/// * [`DebugDotFmt`][crate::dot::DebugDotFmt] -- format each type parameter
/// with its `std::fmt::Debug` implementation.
///
/// * [`DisplayDotFmt`][crate::dot::DisplayDotFmt] -- format each type parameter
/// with its `std::fmt::Display` implementation.
///
/// You can also implement this trait yourself if your type parameters don't
/// implement `Debug` or `Display`, or if you want to format them in some other
/// way.
pub trait DotFmt<TAlphabet, TState, TOutput> {
/// Format a transition edge: `from ---input---> to`.
///
/// This will be inside an [HTML
/// label](https://www.graphviz.org/doc/info/shapes.html#html), so you may
/// use balanced HTML tags.
fn fmt_transition(
&self,
w: &mut impl Write,
from: Option<&TState>,
input: &TAlphabet,
to: Option<&TState>,
) -> io::Result<()>;
/// Format the custom data associated with a state.
///
/// This will be inside an [HTML
/// label](https://www.graphviz.org/doc/info/shapes.html#html), so you may
/// use balanced HTML tags.
fn fmt_state(&self, w: &mut impl Write, state: &TState) -> io::Result<()>;
/// Format a transition's output or the final output of a final state.
///
/// This will be inside an [HTML
/// label](https://www.graphviz.org/doc/info/shapes.html#html), so you may
/// use balanced HTML tags.
fn fmt_output(&self, w: &mut impl Write, output: &TOutput) -> io::Result<()>;
}
impl<TAlphabet, TState, TOutput> Automaton<TAlphabet, TState, TOutput>
where
TAlphabet: Clone + Eq + Hash + Ord,
TState: Clone + Eq + Hash,
TOutput: Output,
{
/// Write this `Automaton` out as a [GraphViz
/// Dot](https://graphviz.gitlab.io/_pages/pdf/dotguide.pdf) file at the
/// given path.
///
/// The `formatter` parameter controls how `TAlphabet`, `TState`, and
/// `TOutput` are rendered. See the [`DotFmt`][crate::dot::DotFmt] trait for
/// details.
///
/// **This method only exists when the `"dot"` cargo feature is enabled.**
pub fn write_dot_file(
&self,
formatter: &impl DotFmt<TAlphabet, TState, TOutput>,
path: impl AsRef<Path>,
) -> io::Result<()> {
let mut file = fs::File::create(path)?;
self.write_dot(formatter, &mut file)?;
Ok(())
}
/// Write this `Automaton` out to the given write-able as a [GraphViz
/// Dot](https://graphviz.gitlab.io/_pages/pdf/dotguide.pdf) file.
///
/// The `formatter` parameter controls how `TAlphabet`, `TState`, and
/// `TOutput` are rendered. See the [`DotFmt`][crate::dot::DotFmt] trait for
/// details.
///
/// **This method only exists when the `"dot"` cargo feature is enabled.**
pub fn write_dot(
&self,
formatter: &impl DotFmt<TAlphabet, TState, TOutput>,
w: &mut impl Write,
) -> io::Result<()> {
writeln!(w, "digraph {{")?;
writeln!(w, " rankdir = \"LR\";")?;
writeln!(w, " nodesep = 2;")?;
// Fake state for the incoming arrow to the start state.
writeln!(w, " \"\" [shape = none];")?;
// Each state, its associated custom data, and its final output.
for (i, state_data) in self.state_data.iter().enumerate() {
write!(
w,
r#" state_{i} [shape = {shape}, label = <<table border="0"><tr><td cellpadding="5">{i}</td></tr><tr><td cellpadding="5">"#,
i = i,
shape = if self.final_states.contains_key(&State(i as u32)) {
"doublecircle"
} else {
"circle"
}
)?;
if let Some(state_data) = state_data {
formatter.fmt_state(w, state_data)?;
} else {
write!(w, "(no state data)")?;
}
write!(w, "</td></tr>")?;
if let Some(final_output) = self.final_states.get(&State(i as u32)) {
write!(w, r#"<tr><td cellpadding="5" align="left">"#)?;
formatter.fmt_output(w, final_output)?;
write!(w, "</td></tr>")?;
}
writeln!(w, "</table>>];")?;
}
// Fake transition to the start state.
writeln!(w, r#" "" -> state_{};"#, self.start_state.0)?;
// Transitions between states and their outputs.
for (from, transitions) in self.transitions.iter().enumerate() {
for (input, (to, output)) in transitions {
write!(
w,
r#" state_{from} -> state_{to} [label = <<table border="0"><tr><td cellpadding="5" align="left">Input:</td><td cellpadding="5" align="left">"#,
from = from,
to = to.0,
)?;
formatter.fmt_transition(
w,
self.state_data[from].as_ref(),
input,
self.state_data[to.0 as usize].as_ref(),
)?;
write!(
w,
r#"</td></tr><tr><td cellpadding="5" align="left">Output:</td><td cellpadding="5" align="left">"#,
)?;
formatter.fmt_output(w, output)?;
writeln!(w, "</td></tr></table>>];")?;
}
}
writeln!(w, "}}")?;
Ok(())
}
}
/// Format an `Automaton`'s `TAlphabet`, `TState`, and `TOutput` with their
/// `std::fmt::Debug` implementations.
#[derive(Debug)]
pub struct DebugDotFmt;
impl<TAlphabet, TState, TOutput> DotFmt<TAlphabet, TState, TOutput> for DebugDotFmt
where
TAlphabet: Debug,
TState: Debug,
TOutput: Debug,
{
fn fmt_transition(
&self,
w: &mut impl Write,
_from: Option<&TState>,
input: &TAlphabet,
_to: Option<&TState>,
) -> io::Result<()> {
write!(w, r#"<font face="monospace">{:?}</font>"#, input)
}
fn fmt_state(&self, w: &mut impl Write, state: &TState) -> io::Result<()> {
write!(w, r#"<font face="monospace">{:?}</font>"#, state)
}
fn fmt_output(&self, w: &mut impl Write, output: &TOutput) -> io::Result<()> {
write!(w, r#"<font face="monospace">{:?}</font>"#, output)
}
}
/// Format an `Automaton`'s `TAlphabet`, `TState`, and `TOutput` with their
/// `std::fmt::Display` implementations.
#[derive(Debug)]
pub struct DisplayDotFmt;
impl<TAlphabet, TState, TOutput> DotFmt<TAlphabet, TState, TOutput> for DisplayDotFmt
where
TAlphabet: Display,
TState: Display,
TOutput: Display,
{
fn fmt_transition(
&self,
w: &mut impl Write,
_from: Option<&TState>,
input: &TAlphabet,
_to: Option<&TState>,
) -> io::Result<()> {
write!(w, "{}", input)
}
fn fmt_state(&self, w: &mut impl Write, state: &TState) -> io::Result<()> {
write!(w, "{}", state)
}
fn fmt_output(&self, w: &mut impl Write, output: &TOutput) -> io::Result<()> {
write!(w, "{}", output)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::Builder;
#[test]
fn test_write_dot() {
let mut builder = Builder::<char, (), u64>::new();
// Insert "mon" -> 1
let mut insertion = builder.insert();
insertion.next('m', 1).next('o', 0).next('n', 0);
insertion.finish();
// Insert "sat" -> 6
let mut insertion = builder.insert();
insertion.next('s', 6).next('a', 0).next('t', 0);
insertion.finish();
// Insert "sun" -> 0
let mut insertion = builder.insert();
insertion.next('s', 0).next('u', 0).next('n', 0);
insertion.finish();
let automata = builder.finish();
let expected = r#"
digraph {
rankdir = "LR";
nodesep = 2;
"" [shape = none];
state_0 [shape = doublecircle, label = <<table border="0"><tr><td cellpadding="5">0</td></tr><tr><td cellpadding="5">(no state data)</td></tr><tr><td cellpadding="5" align="left"><font face="monospace">0</font></td></tr></table>>];
state_1 [shape = circle, label = <<table border="0"><tr><td cellpadding="5">1</td></tr><tr><td cellpadding="5">(no state data)</td></tr></table>>];
state_2 [shape = circle, label = <<table border="0"><tr><td cellpadding="5">2</td></tr><tr><td cellpadding="5">(no state data)</td></tr></table>>];
state_3 [shape = circle, label = <<table border="0"><tr><td cellpadding="5">3</td></tr><tr><td cellpadding="5">(no state data)</td></tr></table>>];
state_4 [shape = circle, label = <<table border="0"><tr><td cellpadding="5">4</td></tr><tr><td cellpadding="5">(no state data)</td></tr></table>>];
state_5 [shape = circle, label = <<table border="0"><tr><td cellpadding="5">5</td></tr><tr><td cellpadding="5">(no state data)</td></tr></table>>];
"" -> state_5;
state_1 -> state_0 [label = <<table border="0"><tr><td cellpadding="5" align="left">Input:</td><td cellpadding="5" align="left"><font face="monospace">'n'</font></td></tr><tr><td cellpadding="5" align="left">Output:</td><td cellpadding="5" align="left"><font face="monospace">0</font></td></tr></table>>];
state_2 -> state_1 [label = <<table border="0"><tr><td cellpadding="5" align="left">Input:</td><td cellpadding="5" align="left"><font face="monospace">'o'</font></td></tr><tr><td cellpadding="5" align="left">Output:</td><td cellpadding="5" align="left"><font face="monospace">0</font></td></tr></table>>];
state_3 -> state_0 [label = <<table border="0"><tr><td cellpadding="5" align="left">Input:</td><td cellpadding="5" align="left"><font face="monospace">'t'</font></td></tr><tr><td cellpadding="5" align="left">Output:</td><td cellpadding="5" align="left"><font face="monospace">0</font></td></tr></table>>];
state_4 -> state_3 [label = <<table border="0"><tr><td cellpadding="5" align="left">Input:</td><td cellpadding="5" align="left"><font face="monospace">'a'</font></td></tr><tr><td cellpadding="5" align="left">Output:</td><td cellpadding="5" align="left"><font face="monospace">6</font></td></tr></table>>];
state_4 -> state_1 [label = <<table border="0"><tr><td cellpadding="5" align="left">Input:</td><td cellpadding="5" align="left"><font face="monospace">'u'</font></td></tr><tr><td cellpadding="5" align="left">Output:</td><td cellpadding="5" align="left"><font face="monospace">0</font></td></tr></table>>];
state_5 -> state_2 [label = <<table border="0"><tr><td cellpadding="5" align="left">Input:</td><td cellpadding="5" align="left"><font face="monospace">'m'</font></td></tr><tr><td cellpadding="5" align="left">Output:</td><td cellpadding="5" align="left"><font face="monospace">1</font></td></tr></table>>];
state_5 -> state_4 [label = <<table border="0"><tr><td cellpadding="5" align="left">Input:</td><td cellpadding="5" align="left"><font face="monospace">'s'</font></td></tr><tr><td cellpadding="5" align="left">Output:</td><td cellpadding="5" align="left"><font face="monospace">0</font></td></tr></table>>];
}
"#;
let mut buf = vec![];
automata.write_dot(&DebugDotFmt, &mut buf).unwrap();
let actual = String::from_utf8(buf).unwrap();
eprintln!("{}", actual);
assert_eq!(expected.trim(), actual.trim());
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,93 @@
use crate::Output;
use std::cmp;
use std::hash::Hash;
impl Output for u64 {
fn empty() -> Self {
0
}
fn prefix(a: &Self, b: &Self) -> Self {
cmp::min(*a, *b)
}
fn difference(a: &Self, b: &Self) -> Self {
a - b
}
fn concat(a: &Self, b: &Self) -> Self {
a + b
}
}
impl<T> Output for Vec<T>
where
T: Clone + Eq + Hash,
{
fn empty() -> Self {
vec![]
}
fn is_empty(&self) -> bool {
self.is_empty()
}
fn prefix(a: &Self, b: &Self) -> Self {
a.iter()
.cloned()
.zip(b.iter().cloned())
.take_while(|(a, b)| a == b)
.map(|(a, _)| a)
.collect()
}
fn difference(a: &Self, b: &Self) -> Self {
let i = a
.iter()
.zip(b.iter())
.position(|(a, b)| a != b)
.unwrap_or(cmp::min(a.len(), b.len()));
a[i..].to_vec()
}
fn concat(a: &Self, b: &Self) -> Self {
let mut c = a.clone();
c.extend(b.iter().cloned());
c
}
}
#[cfg(test)]
mod tests {
use crate::Output;
use std::fmt::Debug;
// Assert the laws that `Output` requires for correctness. `a` and `b`
// should be two different instances of an `Output` type.
fn assert_laws<O>(a: O, b: O)
where
O: Clone + Debug + Output,
{
// Law 1
assert_eq!(O::concat(&O::empty(), &a), a.clone());
// Law 2
assert_eq!(O::prefix(&b, &a), O::prefix(&a, &b));
// Law 3
assert_eq!(O::prefix(&O::empty(), &a), O::empty());
// Law 4
assert_eq!(O::difference(&O::concat(&a, &b), &a), b);
}
#[test]
fn impl_for_u64() {
assert_laws(3, 5);
}
#[test]
fn impl_for_vec() {
assert_laws(vec![0, 1, 2, 3], vec![0, 2, 4, 6]);
}
}

View File

@@ -0,0 +1,195 @@
//! `serde::Serialize` and `serde::Deserialize` implementations for `Automaton`.
//!
//! Rather than prefix each serialized field with which field it is, we always
//! serialize fields in alphabetical order. Make sure to maintain this if you
//! add or remove fields!
//!
//! Each time you add/remove a field, or change serialization in any other way,
//! make sure to bump `SERIALIZATION_VERSION`.
use crate::{Automaton, Output, State};
use serde::{
de::{self, Deserializer, SeqAccess, Visitor},
ser::SerializeTupleStruct,
Deserialize, Serialize, Serializer,
};
use std::collections::BTreeMap;
use std::fmt;
use std::hash::Hash;
use std::marker::PhantomData;
const SERIALIZATION_VERSION: u32 = 1;
impl Serialize for State {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
serializer.serialize_u32(self.0)
}
}
impl<'de> Deserialize<'de> for State {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
Ok(State(deserializer.deserialize_u32(U32Visitor)?))
}
}
struct U32Visitor;
impl<'de> Visitor<'de> for U32Visitor {
type Value = u32;
fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
f.write_str("an integer between `0` and `2^32 - 1`")
}
fn visit_u8<E>(self, value: u8) -> Result<Self::Value, E>
where
E: de::Error,
{
Ok(u32::from(value))
}
fn visit_u32<E>(self, value: u32) -> Result<Self::Value, E>
where
E: de::Error,
{
Ok(value)
}
fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E>
where
E: de::Error,
{
use std::u32;
if value <= u64::from(u32::MAX) {
Ok(value as u32)
} else {
Err(E::custom(format!("u32 out of range: {}", value)))
}
}
}
impl<TAlphabet, TState, TOutput> Serialize for Automaton<TAlphabet, TState, TOutput>
where
TAlphabet: Serialize + Clone + Eq + Hash + Ord,
TState: Serialize + Clone + Eq + Hash,
TOutput: Serialize + Output,
{
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
let Automaton {
final_states,
start_state,
state_data,
transitions,
} = self;
let mut s = serializer.serialize_tuple_struct("Automaton", 5)?;
s.serialize_field(&SERIALIZATION_VERSION)?;
s.serialize_field(final_states)?;
s.serialize_field(start_state)?;
s.serialize_field(state_data)?;
s.serialize_field(transitions)?;
s.end()
}
}
impl<'de, TAlphabet, TState, TOutput> Deserialize<'de> for Automaton<TAlphabet, TState, TOutput>
where
TAlphabet: 'de + Deserialize<'de> + Clone + Eq + Hash + Ord,
TState: 'de + Deserialize<'de> + Clone + Eq + Hash,
TOutput: 'de + Deserialize<'de> + Output,
{
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
deserializer.deserialize_tuple_struct(
"Automaton",
5,
AutomatonVisitor {
phantom: PhantomData,
},
)
}
}
struct AutomatonVisitor<'de, TAlphabet, TState, TOutput>
where
TAlphabet: 'de + Deserialize<'de> + Clone + Eq + Hash + Ord,
TState: 'de + Deserialize<'de> + Clone + Eq + Hash,
TOutput: 'de + Deserialize<'de> + Output,
{
phantom: PhantomData<&'de (TAlphabet, TState, TOutput)>,
}
impl<'de, TAlphabet, TState, TOutput> Visitor<'de>
for AutomatonVisitor<'de, TAlphabet, TState, TOutput>
where
TAlphabet: 'de + Deserialize<'de> + Clone + Eq + Hash + Ord,
TState: 'de + Deserialize<'de> + Clone + Eq + Hash,
TOutput: 'de + Deserialize<'de> + Output,
{
type Value = Automaton<TAlphabet, TState, TOutput>;
fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
f.write_str("Automaton")
}
fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
where
A: SeqAccess<'de>,
{
match seq.next_element::<u32>()? {
Some(v) if v == SERIALIZATION_VERSION => {}
Some(v) => {
return Err(de::Error::invalid_value(
de::Unexpected::Unsigned(v as u64),
&self,
));
}
None => return Err(de::Error::invalid_length(0, &"Automaton expects 5 elements")),
}
let final_states = match seq.next_element::<BTreeMap<State, TOutput>>()? {
Some(x) => x,
None => return Err(de::Error::invalid_length(1, &"Automaton expects 5 elements")),
};
let start_state = match seq.next_element::<State>()? {
Some(x) => x,
None => return Err(de::Error::invalid_length(2, &"Automaton expects 5 elements")),
};
let state_data = match seq.next_element::<Vec<Option<TState>>>()? {
Some(x) => x,
None => return Err(de::Error::invalid_length(3, &"Automaton expects 5 elements")),
};
let transitions = match seq.next_element::<Vec<BTreeMap<TAlphabet, (State, TOutput)>>>()? {
Some(x) => x,
None => return Err(de::Error::invalid_length(4, &"Automaton expects 5 elements")),
};
let automata = Automaton {
final_states,
start_state,
state_data,
transitions,
};
// Ensure that the deserialized automata is well-formed.
automata
.check_representation()
.map_err(|msg| de::Error::custom(msg))?;
Ok(automata)
}
}