chore: generate template dynamically for specific enabled plugins

[pkosiec]

First phase of the dynamic template, that works with analytics and/or lakebase plugins.

Resolves https://databricks.atlassian.net/browse/LKB-9681

Video

lakebase-template.mp4

All PRs

1. CLI: Display plugin setup instructions after apps init flow, remove empty files after template rendering cli#4549
2. feat(lakebase): add ability to lookup DB username with API #123
3. feat(appkit): introduce Lakebase plugin #126
4. chore: generate template dynamically for specific enabled plugins #127

Scope

We agreed on the following scope:

1. remove harcoded paths on CLI - maybe just trigger an npm command as a part of the template at the end (if necessary, let's see)
2. from the template, I'll use Go templates to conditionally add/remove files content

[MarioCadenas]

since the server is already defined as "requiredByTemplate" it will be added automatically, we don't have to hard code it, what would be nice is to get the config from the manifest and inject it here instead, but I guess that can be added later. But this seems very specific to lakebase so it can extend the routes, maybe we can do this in a different way?

[pkosiec]

yes, it is specific to lakebase for now - possibly other plugins can also extend the routes.

would be nice is to get the config from the manifest and inject it here instead

not sure if this can be done - it's purely template-related, and in this particular template, the Lakebase plugin adds some additional routes. I can imagine that the if will change to check if a plugin from a slice is enabled.

IMO the current approach is the most readable one, but it will definitely evolve over time 👍

[MarioCadenas]

this is all very coupled to each plugin, right now the cli populates the env vars based on the resources from the manifest so we wouldn't need this for analytics.

I wonder if we can make it in a different way for lakebase so we can keep what we had instead of putting ifs all around checking for each plugin

[pkosiec]

Sorry, you're right, we should use .appEnv. I'll change it 👍

env:
{{- if .appEnv}}
{{.appEnv}}
{{- end}}
{{- if .plugins.lakebase}}
  ... # additional, not resource-specific envs, until there is no resource we can use
{{- end}}

But for Lakebase, it is still be manual as I want to ensure there are comments near those envs for guidance. I don't see any better way to define this, unless there's a way on plugin manifest to provide env with additional comment - I don't think so? On the other hand, I don't think it makes sense to build it into the plugin manifest 🤔

Instead of putting ifs all around checking for each plugin

Please keep in mind that we'll do the ifs in the template anyway - at least for now. The more "dynamic" the template is, the more complex the structure and syntax is. In the current scale, IMO it's better to add routes / menu links based on if with a given plugin name, instead of having a slice and registering them dynamically which adds significant amount of complexity.

[MarioCadenas]

I wouldn't do this 😅

[pkosiec]

The issue I was fixing was this one - after build and deploy:

node:internal/modules/esm/resolve:275
    throw new ERR_MODULE_NOT_FOUND(
          ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/python/source_code/dist/server/routes/lakebase/todo-routes' imported from /app/python/source_code/dist/server/server.js
    at finalizeResolution (node:internal/modules/esm/resolve:275:11)
    at moduleResolve (node:internal/modules/esm/resolve:860:10)
    at defaultResolve (node:internal/modules/esm/resolve:984:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:780:12)
    at #cachedDefaultResolve (node:internal/modules/esm/loader:704:25)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:687:38)
    at ModuleLoader.getModuleJobForImport (node:internal/modules/esm/loader:305:38)
    at ModuleJob._link (node:internal/modules/esm/module_job:137:49) {
  code: 'ERR_MODULE_NOT_FOUND',
  url: 'file:///app/python/source_code/dist/server/routes/lakebase/todo-routes'
}

Root Cause
The project has "type": "module" in package.json (ESM mode). tsconfig.shared.json uses "moduleResolution": "bundler" — which is designed for code that goes through a bundler (Vite/webpack) and does not require explicit file extensions. The server code is compiled by TypeScript and run directly by Node.js, not through a bundler.
When TypeScript compiles server.ts with moduleResolution: "bundler", it preserves the bare import './routes/lakebase/todo-routes' (no .js) in the emitted JS. Node.js ESM strictly requires explicit .js extensions on relative imports and has no fallback resolution, so it throws ERR_MODULE_NOT_FOUND.

Do you have any better ideas how to fix it? Do we really need ESM build?

[MarioCadenas]

yeah we should do ESM, I was improving this by adding tsdown to the template, so it should be fixed with that

[MarioCadenas]

I like the idea of making the template even more dynamic, but there are some things we need to polish 😄.
We also need to be very careful when we merge this. The template pinning should be merged before this, your changes on the CLI to use this template, and the CLI released with it, if not we can't merge this as it has a lot of breaking changes on the template

[pkosiec]

The template pinning should be merged before this

Absolutely 👍 CLI pinning goes first, then my CLI PR databricks/cli#4549, then this one.