alexion/dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(xatu): construct lexical scope graphs #93

Closed
alexion wants to merge 2 commits from feat/xatu-scope-graph-construction into feat/xatu-debug
pull from: feat/xatu-scope-graph-construction
merge into: alexion:feat/xatu-debug
Conversation 1 Commits 2 Files Changed 10 +984 -22
10 changed files with 984 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
src/xatu/BUILD
View File

@@ -11,6 +11,7 @@ rust_library(
"rules/manual_inherit.rs", "rules/manual_inherit.rs",
"rules/manual_inherit_from.rs", "rules/manual_inherit_from.rs",
"rules/unquoted_uri.rs", "rules/unquoted_uri.rs",
"scope_graph.rs",
"xatu.rs", "xatu.rs",
], ],
deps = [ deps = [

55
src/xatu/README.md
View File

@@ -1,18 +1,51 @@
# Xatu debug output # Xatu scope graphs
Xatu constructs two representations from one rnix syntax tree.
`parser.rs` owns all rnix access.
`ast.rs` contains the lowered representation for lint rules and the `ParsedFile` container.
`scope_graph.rs` contains the lexical scope graph.
`ParsedFile` holds both representations.
The graph follows the scope, declaration, and reference model in Néron et al., *A Theory of Name Resolution* (2015).
See sections 2.2–2.3 and 4.
Graph construction does not resolve references or order declarations.
## Initial coverage
Construction targets the syntax in `src/dotfiles/registry.nix`:
- Recursive let scopes and simple identifier declarations.
- Lambda scopes, identifier parameters, patterns, defaults, and pattern aliases.
- Plain `inherit` references and declarations in lets.
- References in the source expression of `inherit (expression)`.
- References inside attribute values, dynamic attributes, interpolations, assertions, and other expression children.
Static attribute labels do not introduce lexical declarations or references.
Non-recursive attrsets do not introduce lexical scopes.
Plain `inherit` in a let reads its enclosing scope.
Qualified inheritance reads its source expression in the let scope.
Construction omits `with`, recursive attrsets, and legacy lets, including their subtrees.
It does not declare dotted, quoted, or dynamic let bindings.
Other files can produce incomplete graphs without diagnostics.
No completeness guarantee applies outside the supported subset.
Vector indices identify scopes, declarations, and references within one file.
Each occurrence records its name, scope, and half-open UTF-8 byte range.
Scope ranges locate syntax constructs; parent links define lexical nesting.
An inherited name can have declaration and reference occurrences at the same source range.
The graph does not follow Nix imports or resolve attribute members.
## Debug output
Run `bazel run //src/xatu:xatu -- --debug src/dotfiles/registry.nix` to inspect parser output. Run `bazel run //src/xatu:xatu -- --debug src/dotfiles/registry.nix` to inspect parser output.
The command writes JSON containing file paths and complete `ParsedFile` values: The command writes JSON containing file paths and complete `ParsedFile` values.
- File IDs.
- Lowered rule nodes and tokens.
- Parse errors.
It skips lint rules and leaves the normal analysis report unchanged. It skips lint rules and leaves the normal analysis report unchanged.
It sorts files by path and removes duplicate paths before assigning file IDs. It sorts files by path and removes duplicate paths before assigning file IDs.
Parse errors remain in JSON and produce exit code 2. Parse errors remain in JSON and produce exit code 2.
Read errors go to stderr and produce exit code 2 without JSON output. Read errors go to stderr and produce exit code 2 without JSON output.
Successful parsing produces exit code 0, including files with potential lint findings. Successful construction produces exit code 0, including files with potential lint findings.
The debug format is an inspection interface, not a stable interchange schema. The debug format is an inspection interface, not a stable interchange schema.
## Tests ## Tests
@@ -23,6 +56,10 @@ Existing black-box tests check lint findings.
The test also checks parse errors, read errors, and deterministic file ordering. The test also checks parse errors, read errors, and deterministic file ordering.
It copies Nix fixtures into `TEST_TMPDIR` because discovery skips symlinks, including Bazel runfiles. It copies Nix fixtures into `TEST_TMPDIR` because discovery skips symlinks, including Bazel runfiles.
`clean.nix` places outer let declarations in consumer-first, depth-first dependency order from the entry expression.
`lint_findings.nix` deliberately uses a different order and retains cases for existing lint rules.
No ordering rule exists yet.
Regenerate expected JSON after intentional parser or fixture changes: Regenerate expected JSON after intentional parser or fixture changes:
```bash ```bash
@@ -36,5 +73,5 @@ binary="$(pwd)/bazel-bin/src/xatu/xatu"
) )
``` ```
Review regenerated output before accepting it. Review regenerated graphs before accepting them.
Expected JSON files are generated test artifacts; do not edit them manually. Expected JSON files are generated test artifacts; do not edit them manually.

3
src/xatu/ast.rs
View File

@@ -1,6 +1,6 @@
use serde::Serialize; use serde::Serialize;
use crate::FileId; use crate::{FileId, scope_graph::ScopeGraph};
#[derive(Debug, Serialize)] #[derive(Debug, Serialize)]
pub struct ParsedFile { pub struct ParsedFile {
@@ -8,6 +8,7 @@ pub struct ParsedFile {
pub nodes: Vec<Node>, pub nodes: Vec<Node>,
pub tokens: Vec<Token>, pub tokens: Vec<Token>,
pub errors: Vec<ParseError>, pub errors: Vec<ParseError>,
pub scope_graph: ScopeGraph,
} }
#[derive(Debug, Clone, PartialEq, Eq, Serialize)] #[derive(Debug, Clone, PartialEq, Eq, Serialize)]

135
src/xatu/parser.rs
View File

@@ -1,18 +1,21 @@
use rnix::ast::Attr; use rnix::ast::{Attr, HasEntry};
use rowan::ast::AstNode; use rowan::ast::AstNode;
use crate::scope_graph::{Occurrence, Scope, ScopeGraph};
use crate::{FileId, ast}; use crate::{FileId, ast};
pub fn parse_file(file_id: FileId, source: &str) -> ast::ParsedFile { pub fn parse_file(file_id: FileId, source: &str) -> ast::ParsedFile {
let parse = rnix::Root::parse(source); let parse = rnix::Root::parse(source);
let source_len = source.len(); let source_len = source.len();
let scope_graph = build_scope_graph(parse.syntax());
let (nodes, tokens) = convert_syntax(parse.syntax()); let (nodes, tokens) = convert_syntax(parse.syntax());
ast::ParsedFile { ast::ParsedFile {
file_id, file_id,
nodes, nodes,
tokens, tokens,
scope_graph,
errors: parse errors: parse
.errors() .errors()
.iter() .iter()
@@ -21,6 +24,136 @@ pub fn parse_file(file_id: FileId, source: &str) -> ast::ParsedFile {
} }
} }
fn build_scope_graph(syntax: rnix::SyntaxNode) -> ScopeGraph {
let mut graph = ScopeGraph::default();
graph.scopes.push(Scope {
parent: None,
range: syntax.text_range().into(),
});
collect_scopes(syntax, 0, &mut graph);
graph
}
fn collect_scopes(node: rnix::SyntaxNode, scope: usize, graph: &mut ScopeGraph) {
use rnix::SyntaxKind::*;
match node.kind() {
NODE_LET_IN => {
let binding = rnix::ast::LetIn::cast(node.clone()).unwrap();
let inner = add_scope(&node, scope, graph);
for entry in binding.entries() {
match entry {
rnix::ast::Entry::AttrpathValue(binding) => {
if let Some(path) = binding.attrpath() {
let attrs: Vec<_> = path.attrs().collect();
if let [Attr::Ident(ident)] = attrs.as_slice() {
graph.declarations.push(occurrence(ident, inner));
}
}
collect_scopes(binding.syntax().clone(), inner, graph);
}
rnix::ast::Entry::Inherit(inherit) => {
for attr in inherit.attrs() {
if let Attr::Ident(ident) = attr {
graph.declarations.push(occurrence(&ident, inner));
}
}
// Plain inherit reads the enclosing scope, not the recursive let scope.
let reference_scope = if inherit.from().is_some() {
inner
} else {
scope
};
collect_scopes(inherit.syntax().clone(), reference_scope, graph);
}
}
}
if let Some(body) = binding.body() {
collect_scopes(body.syntax().clone(), inner, graph);
}
}
NODE_LAMBDA => {
let lambda = rnix::ast::Lambda::cast(node.clone()).unwrap();
let inner = add_scope(&node, scope, graph);
if let Some(param) = lambda.param() {
match param {
rnix::ast::Param::IdentParam(param) => {
if let Some(ident) = param.ident() {
graph.declarations.push(occurrence(&ident, inner));
}
}
rnix::ast::Param::Pattern(pattern) => {
for entry in pattern.pat_entries() {
if let Some(ident) = entry.ident() {
graph.declarations.push(occurrence(&ident, inner));
}
if let Some(default) = entry.default() {
collect_scopes(default.syntax().clone(), inner, graph);
}
}
if let Some(ident) = pattern.pat_bind().and_then(|bind| bind.ident()) {
graph.declarations.push(occurrence(&ident, inner));
}
}
}
}
if let Some(body) = lambda.body() {
collect_scopes(body.syntax().clone(), inner, graph);
}
}
NODE_IDENT => {
let ident = rnix::ast::Ident::cast(node).unwrap();
graph.references.push(occurrence(&ident, scope));
}
NODE_ATTRPATH => {
// Static attribute names are labels, not lexical references.
for child in node.children().filter(|child| child.kind() != NODE_IDENT) {
collect_scopes(child, scope, graph);
}
}
NODE_INHERIT => {
let inherit = rnix::ast::Inherit::cast(node).unwrap();
if let Some(from) = inherit.from() {
collect_scopes(from.syntax().clone(), scope, graph);
} else {
for attr in inherit.attrs() {
if let Attr::Ident(ident) = attr {
graph.references.push(occurrence(&ident, scope));
}
}
}
}
NODE_WITH | NODE_LEGACY_LET => {}
NODE_ATTR_SET
if rnix::ast::AttrSet::cast(node.clone())
.unwrap()
.rec_token()
.is_some() => {}
_ => {
for child in node.children() {
collect_scopes(child, scope, graph);
}
}
}
}
fn add_scope(node: &rnix::SyntaxNode, parent: usize, graph: &mut ScopeGraph) -> usize {
let id = graph.scopes.len();
graph.scopes.push(Scope {
parent: Some(parent),
range: node.text_range().into(),
});
id
}
fn occurrence(ident: &rnix::ast::Ident, scope: usize) -> Occurrence {
Occurrence {
name: ident.to_string(),
range: ident.syntax().text_range().into(),
scope,
}
}
fn convert_syntax(syntax: rnix::SyntaxNode) -> (Vec<ast::Node>, Vec<ast::Token>) { fn convert_syntax(syntax: rnix::SyntaxNode) -> (Vec<ast::Node>, Vec<ast::Token>) {
let mut nodes = Vec::new(); let mut nodes = Vec::new();
let mut tokens = Vec::new(); let mut tokens = Vec::new();

27
src/xatu/scope_graph.rs Normal file
View File

@@ -0,0 +1,27 @@
use serde::Serialize;
use crate::ast::SourceRange;
/// Captures lexical binding structure without resolving references.
///
/// Vector indices identify scopes and occurrences within one file.
/// Construction follows Néron et al., *A Theory of Name Resolution* (2015), sections 2.2–2.3 and 4.
#[derive(Debug, Default, Serialize)]
pub struct ScopeGraph {
pub scopes: Vec<Scope>,
pub declarations: Vec<Occurrence>,
pub references: Vec<Occurrence>,
}
#[derive(Debug, Serialize)]
pub struct Scope {
pub parent: Option<usize>,
pub range: SourceRange,
}
#[derive(Debug, Serialize)]
pub struct Occurrence {
pub name: String,
pub range: SourceRange,
pub scope: usize,
}

401
src/xatu/testdata/clean.json vendored
View File

@@ -7,8 +7,26 @@
"nodes": [ "nodes": [
{ {
"range": { "range": {
"start": 85, "start": 374,
"end": 103 "end": 392
},
"data": {
"Binding": {
"key": {
"name": "extra"
},
"value": {
"Ident": {
"name": "normalize"
}
}
}
}
},
{
"range": {
"start": 528,
"end": 546
}, },
"data": { "data": {
"Binding": { "Binding": {
@@ -25,7 +43,384 @@
} }
], ],
"tokens": [], "tokens": [],
"errors": [] "errors": [],
"scope_graph": {
"scopes": [
{
"parent": null,
"range": {
"start": 0,
"end": 691
}
},
{
"parent": 0,
"range": {
"start": 0,
"end": 690
}
},
{
"parent": 1,
"range": {
"start": 10,
"end": 690
}
},
{
"parent": 2,
"range": {
"start": 25,
"end": 213
}
},
{
"parent": 3,
"range": {
"start": 36,
"end": 213
}
},
{
"parent": 2,
"range": {
"start": 229,
"end": 277
}
},
{
"parent": 2,
"range": {
"start": 315,
"end": 394
}
}
],
"declarations": [
{
"name": "pkgs",
"range": {
"start": 2,
"end": 6
},
"scope": 1
},
{
"name": "render",
"range": {
"start": 16,
"end": 22
},
"scope": 2
},
{
"name": "value",
"range": {
"start": 25,
"end": 30
},
"scope": 3
},
{
"name": "normalized",
"range": {
"start": 46,
"end": 56
},
"scope": 4
},
{
"name": "value",
"range": {
"start": 90,
"end": 95
},
"scope": 4
},
{
"name": "prefix",
"range": {
"start": 118,
"end": 124
},
"scope": 4
},
{
"name": "normalize",
"range": {
"start": 217,
"end": 226
},
"scope": 2
},
{
"name": "value",
"range": {
"start": 229,
"end": 234
},
"scope": 5
},
{
"name": "fallback",
"range": {
"start": 281,
"end": 289
},
"scope": 2
},
{
"name": "project",
"range": {
"start": 305,
"end": 312
},
"scope": 2
},
{
"name": "value",
"range": {
"start": 317,
"end": 322
},
"scope": 6
},
{
"name": "normalize",
"range": {
"start": 324,
"end": 333
},
"scope": 6
},
{
"name": "args",
"range": {
"start": 344,
"end": 348
},
"scope": 6
}
],
"references": [
{
"name": "normalize",
"range": {
"start": 59,
"end": 68
},
"scope": 4
},
{
"name": "value",
"range": {
"start": 69,
"end": 74
},
"scope": 4
},
{
"name": "value",
"range": {
"start": 90,
"end": 95
},
"scope": 3
},
{
"name": "pkgs",
"range": {
"start": 112,
"end": 116
},
"scope": 4
},
{
"name": "normalized",
"range": {
"start": 144,
"end": 154
},
"scope": 4
},
{
"name": "null",
"range": {
"start": 158,
"end": 162
},
"scope": 4
},
{
"name": "normalized",
"range": {
"start": 172,
"end": 182
},
"scope": 4
},
{
"name": "prefix",
"range": {
"start": 189,
"end": 195
},
"scope": 4
},
{
"name": "normalized",
"range": {
"start": 198,
"end": 208
},
"scope": 4
},
{
"name": "value",
"range": {
"start": 239,
"end": 244
},
"scope": 5
},
{
"name": "null",
"range": {
"start": 248,
"end": 252
},
"scope": 5
},
{
"name": "fallback",
"range": {
"start": 258,
"end": 266
},
"scope": 5
},
{
"name": "value",
"range": {
"start": 272,
"end": 277
},
"scope": 5
},
{
"name": "value",
"range": {
"start": 336,
"end": 341
},
"scope": 6
},
{
"name": "args",
"range": {
"start": 361,
"end": 365
},
"scope": 6
},
{
"name": "normalize",
"range": {
"start": 382,
"end": 391
},
"scope": 6
},
{
"name": "render",
"range": {
"start": 412,
"end": 418
},
"scope": 2
},
{
"name": "project",
"range": {
"start": 420,
"end": 427
},
"scope": 2
},
{
"name": "pkgs",
"range": {
"start": 520,
"end": 524
},
"scope": 2
},
{
"name": "other",
"range": {
"start": 540,
"end": 545
},
"scope": 2
},
{
"name": "nested",
"range": {
"start": 566,
"end": 572
},
"scope": 2
},
{
"name": "quoted",
"range": {
"start": 595,
"end": 601
},
"scope": 2
},
{
"name": "dynamic",
"range": {
"start": 607,
"end": 614
},
"scope": 2
},
{
"name": "dynamic",
"range": {
"start": 618,
"end": 625
},
"scope": 2
},
{
"name": "pkgs",
"range": {
"start": 636,
"end": 640
},
"scope": 2
},
{
"name": "pkgs",
"range": {
"start": 666,
"end": 670
},
"scope": 2
},
{
"name": "null",
"range": {
"start": 683,
"end": 687
},
"scope": 2
}
]
}
} }
} }
] ]

15
src/xatu/testdata/clean.nix vendored
View File

@@ -1,6 +1,19 @@
{ pkgs }: { pkgs }:
let
render = value:
let
normalized = normalize value;
inherit value;
inherit (pkgs) prefix;
in
assert normalized != null;
{ ${normalized} = "${prefix}${normalized}"; };
normalize = value: if value == null then fallback else value;
fallback = "default";
project = { value, normalize ? value }@args: { inherit (args) value; extra = normalize; };
in
{ {
result = render (project { value = "name"; }).value;
path = ./. + "/foo"; path = ./. + "/foo";
url = "https://example.com"; url = "https://example.com";
inherit pkgs; inherit pkgs;

353
src/xatu/testdata/lint_findings.json vendored
View File

@@ -7,8 +7,26 @@
"nodes": [ "nodes": [
{ {
"range": { "range": {
"start": 4, "start": 173,
"end": 22 "end": 191
},
"data": {
"Binding": {
"key": {
"name": "extra"
},
"value": {
"Ident": {
"name": "normalize"
}
}
}
}
},
{
"range": {
"start": 458,
"end": 476
}, },
"data": { "data": {
"Binding": { "Binding": {
@@ -25,8 +43,8 @@
}, },
{ {
"range": { "range": {
"start": 25, "start": 479,
"end": 42 "end": 496
}, },
"data": { "data": {
"Binding": { "Binding": {
@@ -48,15 +66,336 @@
"tokens": [ "tokens": [
{ {
"range": { "range": {
"start": 51, "start": 505,
"end": 70 "end": 524
}, },
"data": { "data": {
"UnquotedUri": "https://example.com" "UnquotedUri": "https://example.com"
} }
} }
], ],
"errors": [] "errors": [],
"scope_graph": {
"scopes": [
{
"parent": null,
"range": {
"start": 0,
"end": 528
}
},
{
"parent": 0,
"range": {
"start": 0,
"end": 527
}
},
{
"parent": 1,
"range": {
"start": 10,
"end": 527
}
},
{
"parent": 2,
"range": {
"start": 52,
"end": 100
}
},
{
"parent": 2,
"range": {
"start": 114,
"end": 193
}
},
{
"parent": 2,
"range": {
"start": 206,
"end": 394
}
},
{
"parent": 5,
"range": {
"start": 217,
"end": 394
}
}
],
"declarations": [
{
"name": "pkgs",
"range": {
"start": 2,
"end": 6
},
"scope": 1
},
{
"name": "fallback",
"range": {
"start": 16,
"end": 24
},
"scope": 2
},
{
"name": "normalize",
"range": {
"start": 40,
"end": 49
},
"scope": 2
},
{
"name": "value",
"range": {
"start": 52,
"end": 57
},
"scope": 3
},
{
"name": "project",
"range": {
"start": 104,
"end": 111
},
"scope": 2
},
{
"name": "value",
"range": {
"start": 116,
"end": 121
},
"scope": 4
},
{
"name": "normalize",
"range": {
"start": 123,
"end": 132
},
"scope": 4
},
{
"name": "args",
"range": {
"start": 143,
"end": 147
},
"scope": 4
},
{
"name": "render",
"range": {
"start": 197,
"end": 203
},
"scope": 2
},
{
"name": "value",
"range": {
"start": 206,
"end": 211
},
"scope": 5
},
{
"name": "prefix",
"range": {
"start": 242,
"end": 248
},
"scope": 6
},
{
"name": "value",
"range": {
"start": 264,
"end": 269
},
"scope": 6
},
{
"name": "normalized",
"range": {
"start": 277,
"end": 287
},
"scope": 6
}
],
"references": [
{
"name": "value",
"range": {
"start": 62,
"end": 67
},
"scope": 3
},
{
"name": "null",
"range": {
"start": 71,
"end": 75
},
"scope": 3
},
{
"name": "fallback",
"range": {
"start": 81,
"end": 89
},
"scope": 3
},
{
"name": "value",
"range": {
"start": 95,
"end": 100
},
"scope": 3
},
{
"name": "value",
"range": {
"start": 135,
"end": 140
},
"scope": 4
},
{
"name": "args",
"range": {
"start": 160,
"end": 164
},
"scope": 4
},
{
"name": "normalize",
"range": {
"start": 181,
"end": 190
},
"scope": 4
},
{
"name": "pkgs",
"range": {
"start": 236,
"end": 240
},
"scope": 6
},
{
"name": "value",
"range": {
"start": 264,
"end": 269
},
"scope": 5
},
{
"name": "normalize",
"range": {
"start": 290,
"end": 299
},
"scope": 6
},
{
"name": "value",
"range": {
"start": 300,
"end": 305
},
"scope": 6
},
{
"name": "normalized",
"range": {
"start": 325,
"end": 335
},
"scope": 6
},
{
"name": "null",
"range": {
"start": 339,
"end": 343
},
"scope": 6
},
{
"name": "normalized",
"range": {
"start": 353,
"end": 363
},
"scope": 6
},
{
"name": "prefix",
"range": {
"start": 370,
"end": 376
},
"scope": 6
},
{
"name": "normalized",
"range": {
"start": 379,
"end": 389
},
"scope": 6
},
{
"name": "render",
"range": {
"start": 412,
"end": 418
},
"scope": 2
},
{
"name": "project",
"range": {
"start": 420,
"end": 427
},
"scope": 2
},
{
"name": "package",
"range": {
"start": 468,
"end": 475
},
"scope": 2
},
{
"name": "pkgs",
"range": {
"start": 486,
"end": 490
},
"scope": 2
}
]
}
} }
} }
] ]

15
src/xatu/testdata/lint_findings.nix vendored
View File

@@ -1,4 +1,19 @@
{ pkgs }:
let
fallback = "default";
normalize = value: if value == null then fallback else value;
project = { value, normalize ? value }@args: { inherit (args) value; extra = normalize; };
render = value:
let
inherit (pkgs) prefix;
inherit value;
normalized = normalize value;
in
assert normalized != null;
{ ${normalized} = "${prefix}${normalized}"; };
in
{ {
result = render (project { value = "name"; }).value;
package = package; package = package;
from = pkgs.from; from = pkgs.from;
url = https://example.com; url = https://example.com;

1
src/xatu/xatu.rs
View File

@@ -2,6 +2,7 @@ mod ast;
mod file_store; mod file_store;
mod parser; mod parser;
mod rules; mod rules;
mod scope_graph;
pub use file_store::{FileId, FileStore, SourceFile}; pub use file_store::{FileId, FileStore, SourceFile};
pub use rules::{Diagnostic, Finding}; pub use rules::{Diagnostic, Finding};