Forma Toolchain

A DSL-builder and runtime platform: build your own DSL and make it run.

Forma is the generic language and runtime platform under the Meta-Effects project. It is not one fixed business vocabulary. It is a substrate for defining vocabularies: a small Lisp core, a descriptor system for new forms, protocol declarations for the data those forms produce, elaboration rules that lower author-friendly syntax into canonical intermediate representation, and runtime contracts that make the result executable.

The audience is people building a language surface, not only people writing programs in one already-built language. With Forma, you can define a DSL for a domain, implement an instance of that DSL in source files, and get the benefits a normal hand-rolled configuration language rarely gets: parsing, typechecking, structural validation, diagnostics, editor metadata, IR packaging, and backend code generation from the same declared contracts.

The Language SDK is built around an OCaml compiler core for parsing, expansion, lowering, typechecking, sessions, diagnostics, and editor projections. Forma should still feel like a product surface, not an OCaml project: language authors define forms and elaborations, and target emitters consume the checked IR.

1. The Shape#

The engine is split into three concerns.

text

Lisp source  -> reader, macro expansion, typed core  -> descriptor registry  -> elaboration registry  -> protocol-validated IR  -> target emitters

The language engine owns syntax, spans, expansion, evaluation, type inference, sessions, diagnostics, and artifact packaging. Preludes own the vocabulary. A prelude teaches the engine that a form exists, what arguments and slots it accepts, what it contributes to the compile-time environment, and how it elaborates.

That separation is the main idea:

text

engine     = generic compiler machineryprelude    = language designsource     = one instance of that languageIR         = typed contract for downstream targetsbackend    = generator, validator, runtime adapter, or tool surface

2. Lisp as the Meta Surface#

Forma uses S-expressions because a DSL should not need a custom parser for every new construct. Lists, symbols, keywords, vectors, maps, strings, numbers, and booleans are enough to describe a useful surface syntax.

lisp

(service http-api  (:method GET)  (:path "/customers/:id")  (:input CustomerLookup)  (:returns CustomerProfile)  (:auth requires-user))

The syntax is intentionally small. The meaning lives in the active form descriptors. If the loaded prelude defines service, the compiler knows what this form means. If the prelude does not define it, it is just an unknown head with a span and a diagnostic.

3. define-form#

define-form declares the surface contract for one DSL construct. It names the head, positional identifiers, keyword slots, child forms, validation hooks, binding hooks, construction hooks, and result type behavior.

lisp

(define-form service  (:doc "Declares a deployable service endpoint.")  (:phase domain)  (:identifiers    (identifier name Symbol (:declaration true)))  (:slots    (slot method value (:required true))    (slot path value (:required true))    (slot input value)    (slot returns value)    (slot auth value))  (:bindings-fn service/bindings)  (:validate-fn service/validate)  (:construct-fn service/construct)  (:result-type-fn service/result-type))

This is not merely documentation. The descriptor is executable compiler metadata. It tells the parser how to recognize the declaration shape, tells the editor what slots are valid, tells the type path what names are introduced, and tells elaboration which hook owns construction.

The important consequence is that new DSL forms do not require engine changes. The engine sees descriptors. The prelude supplies language semantics.

4. Forms Are More Than Macros#

Macros rewrite source. Forms declare language-level objects.

A macro can make authoring nicer by expanding shorthand into ordinary source. A form descriptor gives the compiler a stable semantic boundary. It can contribute bindings, enforce slot contracts, preserve provenance, choose a result type, and produce IR.

text

macro expansion  source convenience -> source convenienceform elaboration  source declaration -> typed artifact

That distinction is what makes the system toolable. The compiler can answer "what does this file declare?" without evaluating arbitrary user code. An editor can offer completions for the :method and :path slots because service declared them. A backend can consume the service IR without knowing how the author happened to spell the source.

5. define-elaboration#

define-elaboration is the declarative path from a form to an IR object. It binds a form descriptor to a construction hook and then describes the projection from identifiers, slots, child forms, literals, defaults, references, and named primitives into an emitted payload.

lisp

(define-elaboration service-elaboration  (:hook service/construct)  (:form service)  (:kind "Service")  (:result-type "ServiceDef")  (:name name (:identifier name) (:default "anonymous-service"))  (:field name (:identifier name))  (:field route    (:object      (:field method (:slot-symbol method))      (:field path (:slot-string path))))  (:field input (:slot-symbol input))  (:field returns (:slot-symbol returns))  (:field auth (:slot-symbol auth))  (:field loc (:loc)))

The real descriptor vocabulary includes structural sources such as :identifier, :slot-string, :slot-symbol, :slot-string-list, :slot-runtime-expr, :children, :child, :assignments, :literal, :default, :first, :format, :ref, :object, :positional, :primitive, and :loc.

This gives the common case a dataflow shape:

text

input form  -> read identifiers  -> normalize slots  -> recurse into child forms  -> call closed primitives when structural projection is not enough  -> assemble payload  -> attach summary, result type, and provenance

The direction of travel is to keep pure structural lowering in define-elaboration and reserve imperative hooks for genuinely semantic operations.

6. meta-fn#

meta-fn is the compiler escape hatch. It declares a named compile-time function that a form descriptor can call for binding, validation, construction, or result-type inference when declarative elaboration is not expressive enough.

lisp

(meta-fn service/validate  (:kind validate)  (:input FormMetaInput)  (:output ValidationResult)  (:doc "Checks route method and path invariants for service declarations.")  (:body    (let [method (meta/slot-symbol input :method)          path (meta/slot-string input :path)]      (validate/all        (validate/member method [GET POST PATCH DELETE])        (validate/starts-with path "/")))))

A meta-fn runs at compile time against normalized form input. It can inspect slots, look up declarations already loaded into the session, synthesize bindings, call closed compiler helpers, and construct payloads that are difficult to describe as a structural projection.

Use it when the compiler needs semantics, not just shape:

Binding hooks add names or types to the compile-time environment.
Validation hooks enforce invariants that depend on more than one slot or on external declarations.
Construct hooks build IR through imperative logic when a simple field projection is insufficient.
Result-type hooks derive the type produced by a form from its arguments or context.

The boundary is intentional. define-elaboration is preferable when a form can be lowered by reading identifiers, slots, children, defaults, references, and primitives. meta-fn is for the cases that need real analysis: cross-form lookup, type synthesis, peer DSL compilation, custom normalization, or compatibility with a backend-specific contract.

That makes meta-fn powerful, but not the default authoring style. A DSL should put as much of its surface contract as possible in descriptors and protocols, then use meta-fn only where the language actually needs executable compiler behavior.

7. define-protocol#

define-protocol describes the shape of data that crosses a boundary. Protocols can define records, sums, type aliases, imports, literal fields, arrays, references, and schema names. They are used to generate TypeScript types, Effect Schemas, validators, and artifact contracts.

lisp

(define-protocol ServiceProtocol  (record ServiceRoute    (:schema-name ServiceRoute)    (:fields      (:method (:kind literal) (:values [GET POST PATCH DELETE]) (:required true))      (:path (:type string) (:required true))))  (record ServiceDef    (:schema-name ServiceDef)    (:fields      (:kind (:kind literal) (:values [Service]) (:required true))      (:name (:type string) (:required true))      (:route (:type ServiceRoute) (:required true))      (:input (:type string))      (:returns (:type string))      (:auth (:type string)))))

Protocols matter because IR should be a contract, not a JSON convention. Once a protocol exists, generated code and validators can share the same account of what a ServiceDef is. The DSL surface can change while the backend target keeps consuming the same protocol shape, or the protocol can version when the target contract really changes.

8. Implementing an Instance#

After the prelude defines the DSL, authors write ordinary source in that DSL.

lisp

(service customer-profile  (:method GET)  (:path "/customers/:id")  (:input CustomerLookup)  (:returns CustomerProfile)  (:auth requires-user))(service update-customer-status  (:method PATCH)  (:path "/customers/:id/status")  (:input CustomerStatusPatch)  (:returns CustomerProfile)  (:auth requires-admin))

The source is an instance of the language. It is not a pile of inert config. The engine can parse it, typecheck names and expressions against the active environment, validate descriptor contracts, elaborate declarations, and emit artifacts with source spans.

If CustomerStatusPatch is not defined, diagnostics can point at the exact slot value. If :method is invalid or :path is missing, descriptor validation owns the error. If an emitter needs OpenAPI, RPC handlers, test fixtures, or client SDK code, it consumes the typed IR rather than reparsing the source.

9. IR as the Join Point#

The source language should be pleasant for authors. The IR should be precise for machines. Forma keeps those two shapes separate.

text

author DSL  (service update-customer-status ...)elaborated declaration  kind: Service  resultType: ServiceDef  payload:    name: update-customer-status    route: { method: PATCH, path: "/customers/:id/status" }    input: CustomerStatusPatch    returns: CustomerProfile    auth: requires-admin  provenance:    sourceId, span, prelude hashes

Canonical IR is the convergence point for downstream work. A backend should not care whether the declaration came from a compact Lisp form, a generated source file, a graphical authoring tool, or an agent edit. It should consume the same typed declaration envelope.

That is the payoff: each target gets to be simpler because parsing, binding, validation, and source provenance have already happened.

10. Targets#

A Forma DSL can elaborate to multiple targets through IR-consuming emitters.

Type definitions for TypeScript, Effect Schema, JSON Schema, or host-specific validators.
Runtime packages that register handlers, actions, state machines, or queries.
API artifacts such as OpenAPI documents, MCP tool descriptors, RPC contracts, and SDK clients.
Editor services such as completions, hover, definition lookup, document symbols, diagnostics, and formatting.
Testing surfaces such as generated fixtures, golden IR snapshots, contract tests, and malformed payload cases.
Operational artifacts such as Fact Store schemas, migrations, permission matrices, workflow plans, queues, and task definitions.

The elaboration tooling is aimed at practical host targets:

JavaScript and TypeScript for web, Node, SDKs, schemas, APIs, and agent tools.
Python for data, scripting, workflow integration, and service environments.
Rust for high-integrity runtimes, CLIs, embedded tools, and systems integrations.

The runtime side should support deploy targets for:

Node for standard server deployments and integrated web/API processes.
Bun for fast local and edge-adjacent deployments where it fits.
Cloudflare Durable Objects for actor-like state, durable coordination, and edge deployment.

The same source can feed several targets because the compiler does not lower directly from surface syntax to one runtime. It lowers source to IR, then lets targets consume that IR according to explicit protocols, including the Fact Store that persists runtime assertions.

11. Why This Is Valuable#

Most internal DSLs start as a parser plus ad hoc JSON. They become expensive when teams need better errors, editor support, versioned contracts, generated clients, multiple runtimes, or agents that can edit source safely.

Forma makes the language definition itself the reusable asset.

The parser is generic. New forms do not need a new grammar.
The surface is declared. Slots, identifiers, repeated children, docs, and examples live beside the form definition.
Validation is structured. Descriptor errors and type errors carry spans.
Elaboration is inspectable. Structural lowering is authored as data, not hidden in a private compiler function.
Meta hooks are explicit. Imperative compiler behavior is named, typed by kind, and attached to form descriptors.
Protocols are explicit. IR payloads can be validated, generated, versioned, and shared across packages.
Backends are decoupled. Codegen targets consume typed IR instead of duplicating source semantics.
Agents get handles. They can inspect declared forms, follow references, edit instances, and run the same compiler checks humans use.

The result is not "Lisp for Lisp's sake." It is a way to make a domain language real enough to compile.

12. One-Shot Summary#

Forma is:

a small Lisp substrate for source, macros, and compile-time hooks;
define-form for declaring DSL syntax and compiler-visible form contracts;
define-elaboration for lowering forms into typed IR payloads;
meta-fn for named compile-time hooks when binding, validation, construction, or result-type inference needs executable logic;
define-protocol for specifying the data contracts those payloads must satisfy;
sessions and preludes for loading a language package into a compiler environment;
source spans and diagnostics for human and agent authoring;
canonical IR as the stable boundary between language design and targets;
emitters for validation, code generation, runtime adapters, editor services, and SDKs.

The practical workflow is: design a DSL, declare its forms and protocols, define elaboration into IR, write instances of the DSL, then let the compiler give you validation, provenance, Fact Store mappings, and target artifacts from one source of truth.