Ensure refs in tsconfig files are synced with internal deps#8384
Ensure refs in tsconfig files are synced with internal deps#8384
Conversation
We use project references to tell TypeScript and other tools about the
structure of the monorepo. Both the root- and package-level tsconfig
files have a `references` field which is used to link packages together.
All packages that we want to build and publish need to be present in
`references` within the root `tsconfig.json` and `tsconfig.build.json`
files, and the `references` field in both root-level and package-level
tsconfig files need to list all internal dependencies.
However, manually keeping all of these references up to date as packages
are added or dependencies are updated is a pain. There are cases where
packages were not published or type errors occurred because the
references were not correctly kept in sync.
We don't need to do this manual work. We can infer root references by
scanning workspaces, and we can infer references for an individual
package by scanning that package's `dependencies` for other workspace
packages.
This commit adds a script that can be run on both the root workspace and
child workspaces to check and/or regenerate the `references` field for
each tsconfig file so that it is perfectly kept in sync with
dependencies throughout the monorepo. There are also some package
scripts for use:
- Root level
- **`lint:tsconfigs`:** Validates the root `tsconfig.json` and
`tsconfig.build.json`, erroring if any are out of sync.
- **`lint:tsconfigs:all`:** Validates all `tsconfig.json` and
`tsconfig.build.json` across the monorepo, erroring if any are out
of sync.
- **`lint:tsconfigs:fix`:** Regenerates the root `tsconfig.json` and
`tsconfig.build.json`.
- **`lint:tsconfigs:fix:all`:** Regenerates `tsconfig.json` and
`tsconfig.build.json` files across the monorepo.
- Package level
- **`lint:tsconfigs`:** Validates `tsconfig.json` and
`tsconfig.build.json` in the package, erroring if any are out of
sync.
- **`lint:tsconfigs:fix`:** Regenerates `tsconfig.json` and
`tsconfig.build.json` in the package.
As the names indicate, tsconfig file validation is also a part of the
lint pipeline, so CI will now fail if any files are out of date.
Finally, it's worth noting that there are some existing solutions in the
TypeScript community, some of which are documented in [this issue][1],
but all of which were rejected for various reasons:
- **[`typescript-monorepo-toolkit`][2]:** Doesn't update references for
individual packages based on `dependencies`
- **[`update-ts-references`][3]:** Works, but re-sorts all references,
and drops the leading `./` from references in root tsconfig files;
also, generated files fail Prettier validation
- **[`@monorepo-utils/workspaces-to-typescript-project-references`][4]:**
Works, but re-sorts all references, and drops the leading `./` from
references in root tsconfig files; also generated files fail Prettier
validation
Essentially, by building our own script, we get to control the exact
changes that are made to tsconfig files, and we get to run all files
through Prettier so that engineers do not have to reformat them
manually.
[1]: microsoft/TypeScript#25376
[2]: https://github.com/Bnaya/typescript-monorepo-toolkit
[3]: https://github.com/eBayClassifiedsGroup/update-ts-references
[4]: https://github.com/azu/monorepo-utils/tree/master/packages/@monorepo-utils/workspaces-to-typescript-project-references
2929cc5 to
6cb4611
Compare
| "path": "../messenger/tsconfig.build.json" | ||
| }, | ||
| { | ||
| "path": "../controller-utils/tsconfig.build.json" |
There was a problem hiding this comment.
This dependency is present in devDependencies but was missing as a reference.
| "path": "../messenger" | ||
| }, | ||
| { | ||
| "path": "../controller-utils" |
There was a problem hiding this comment.
This dependency is present in devDependencies but was missing as a reference.
| "path": "../preferences-controller/tsconfig.build.json" | ||
| }, | ||
| { | ||
| "path": "../controller-utils/tsconfig.build.json" |
There was a problem hiding this comment.
All of these dependencies are present in dependencies but were missing as references.
| "path": "../preferences-controller" | ||
| }, | ||
| { | ||
| "path": "../controller-utils" |
There was a problem hiding this comment.
All of these dependencies are present in dependencies but were missing as references.
| "path": "../phishing-controller/tsconfig.build.json" | ||
| }, | ||
| { | ||
| "path": "../multichain-account-service/tsconfig.build.json" |
There was a problem hiding this comment.
These dependencies are present in dependencies but were missing as references.
| "path": "../transaction-controller/tsconfig.build.json" | ||
| }, | ||
| { | ||
| "path": "../messenger/tsconfig.build.json" |
| "path": "../messenger/tsconfig.build.json" | ||
| }, | ||
| { | ||
| "path": "../approval-controller/tsconfig.build.json" |
| "path": "../approval-controller/tsconfig.build.json" | ||
| }, | ||
| { | ||
| "path": "../eth-block-tracker/tsconfig.build.json" |
| "path": "./packages/user-operation-controller" | ||
| }, | ||
| { | ||
| "path": "./packages/eip-5792-middleware" |
There was a problem hiding this comment.
Not sure how these were never added to the root tsconfig 🤔
| '**/tests/**/*.{js,ts}', | ||
| 'scripts/*.ts', | ||
| 'scripts/create-package/**/*.ts', | ||
| 'scripts/**/*.ts', |
There was a problem hiding this comment.
We've added a new subdirectory to scripts so rather than adding an extra entry here we can consolidate them all into one glob.
|
No dependency changes detected. Learn more about Socket for GitHub. 👍 No dependency changes detected in pull request |
| }, | ||
| { | ||
| "path": "../messenger/tsconfig.build.json" | ||
| "path": "../controller-utils/tsconfig.build.json" |
There was a problem hiding this comment.
Not entirely a new dependency, but it should have been added before (see above).
| "@ethereumjs/util": "^9.1.0", | ||
| "@metamask/base-controller": "^9.0.1", | ||
| "@metamask/browser-passworder": "^6.0.0", | ||
| "@metamask/controller-utils": "^11.20.0", |
There was a problem hiding this comment.
Adding this dependency was necessary to avoid a type error. This should have been added — and the type error should have been present — all along, but for some reason only popped up now. It's possible that with the new changes on this branch, keyring-controller was getting built before controller-utils; if so, this corrects it.
| }, | ||
| "devDependencies": { | ||
| "@metamask/auto-changelog": "^3.4.4", | ||
| "@metamask/network-controller": "^30.0.1", |
There was a problem hiding this comment.
It turns out that there has been a circular dependency between network-controller and eth-json-rpc-middleware ever since the latter package was moved from the eth-json-rpc-middleware repo, and the problem didn't pop up until now.
network-controller is used in a test to confirm that a type that eth-json-rpc-middleware copies from network-controller is compatible with the same type from network-controller itself. I wasn't sure that this was necessary because now that eth-json-rpc-middleware is in this monorepo, if there are any type incompatibilities between types, they should show up locally (and any breaking changes to types should be spotted during development or at least during review). So I opted to remove the test file and thus the dependency on network-controller.
| "path": "../json-rpc-engine" | ||
| }, | ||
| { | ||
| "path": "../network-controller" |
|
@SocketSecurity ignore npm/@metamask/preferences-controller@23.1.0 This is our package. |
|
Adding |
| }, | ||
| "dependencies": { | ||
| "@metamask/messenger": "^1.1.1", | ||
| "@metamask/preferences-controller": "^23.1.0", |
There was a problem hiding this comment.
Added to avoid a new type error. Similar to #8384 (comment).
|
Curious if our team should be co-codeowners of |
I do believe we should |
Explanation
We use project references to tell TypeScript and other tools about the structure of the monorepo. Both the root- and package-level tsconfig files have a
referencesfield which is used to link packages together. All packages that we want to build and publish need to be present inreferenceswithin the roottsconfig.jsonandtsconfig.build.jsonfiles, and thereferencesfield in both root-level and package-level tsconfig files need to list all internal dependencies.However, it's a pain to keep references up to date, especially as as packages are added or dependencies are updated. There are cases in the past, in fact, where packages were not published or type errors occurred because the references were not correctly kept in sync.
We don't need to do this manual work. We can infer root references by scanning workspaces, and we can infer references for an individual package by scanning that package's
dependenciesfor other workspace packages.This commit adds a script that can be run on both the root workspace and child workspaces to check and/or regenerate the
referencesfield for each tsconfig file so that it is kept in sync with dependencies throughout the monorepo. There are also new package scripts for use:lint:tsconfigs: Validates the roottsconfig.jsonandtsconfig.build.json, erroring if any are out of sync.lint:tsconfigs:all: Validates alltsconfig.jsonandtsconfig.build.jsonacross the monorepo, erroring if any are out of sync.lint:tsconfigs:fix: Regenerates the roottsconfig.jsonandtsconfig.build.json.lint:tsconfigs:fix:all: Regeneratestsconfig.jsonandtsconfig.build.jsonfiles across the monorepo.lint:tsconfigs: Validatestsconfig.jsonandtsconfig.build.jsonin the package, erroring if any are out of sync.lint:tsconfigs:fix: Regeneratestsconfig.jsonandtsconfig.build.jsonin the package.As the names indicate, tsconfig file validation is also a part of the lint pipeline, so CI will now fail if any files are out of date. To make sure this doesn't happen in the future, this commit also corrects tsconfig files across the board to add missing references or remove extra references.
Finally, it's worth noting that there are some existing solutions for syncing references within the TypeScript community, some of which are documented in this issue. However, I reviewed them and rejected them for various reasons:
typescript-monorepo-toolkit: Doesn't update references for individual packages based ondependenciesupdate-ts-references: Works, but re-sorts all references, and drops the leading./from references in root tsconfig files; also, generated files fail Prettier validation@monorepo-utils/workspaces-to-typescript-project-references: Works, but re-sorts all references, and drops the leading./from references in root tsconfig files; also generated files fail Prettier validationMy thought is that by building our own script, we get to control the exact changes that are made to tsconfig files, and we get to run all files through Prettier so that engineers do not have to reformat them manually.
References
Closes #966.
Checklist
Note
Medium Risk
Medium risk because it touches build tooling/TypeScript project references across many packages, which can break CI builds or publishing if reference inference is wrong, though runtime code changes are minimal.
Overview
Adds a new
scripts/lint-tsconfigstool and wires it into the root lint pipeline and CI (lint:tsconfigs:all), with--fixvariants to regeneratetsconfigreferences.Updates many package
tsconfig.json/tsconfig.build.jsonfiles to syncreferenceswith internal dependencies (including adding/removing specific refs), and adds per-packagelint:tsconfigsscripts (plustsxdevDependency where needed) so workspaces can validate/fix locally.Includes a few related cleanups: broadens ESLint’s node config to cover
scripts/**/*.ts, tweaks README dependency graph entries, removes anetwork-controllertype-compat test frometh-json-rpc-middleware, and adds explicit deps in a couple packages (e.g.,@metamask/preferences-controllerineip-5792-middleware,@metamask/controller-utilsinkeyring-controller).Reviewed by Cursor Bugbot for commit f6f4a91. Bugbot is set up for automated code reviews on this repo. Configure here.