Adding SQLite support after the fact needs better documentation and doesn't work properly

[calston]

Description

Steps to Reproduce the Problem

1. $ go get -u -v -tags sqlite github.com/gobuffalo/buffalo/buffalo
2. $ ~/go/bin/buffalo new test && cd test
3. `$ ~/go/bin/buffalo pop create -a
   v4.10.0

[POP] 2019/04/15 16:05:58 warn - unable to load connection development: could not create new connection: sqlite3 support was not compiled into the binary`

Expected Behavior

[POP] 2019/04/15 16:05:58 info - hey cool things worked and this trivial problem was tested in CI

Actual Behavior

Sqlite3 process per the documentation does not result in sqlite3 support

Info

-> Go: Checking installation
✓ The `go` executable was found on your system at: /usr/local/go/bin/go

-> Go: Checking minimum version requirements
✓ Your version of Go, 1.12.3, meets the minimum requirements.

-> Go: Checking GOPATH
✓ You appear to be operating inside of your GOPATH.

-> Go: Checking Package Management
⚠ You do not appear to be using a package management system.

It is strongly suggested that you use one of the following package management systems:

* Go Modules (Recommended) - https://gobuffalo.io/en/docs/gomods
* Dep - https://github.com/golang/dep

For help setting up your Go environment please follow the instructions for you platform at:

https://www.gopherguides.com/courses/preparing-your-environment-for-go-development

-> Go: Checking PATH
✘ Your PATH (/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:~/Library/Python/2.7/bin) does not contain /Users/<CUT>/go/bin.

Without /Users/<CUT>/go/bin in your `PATH` any Go executables can not be run globally.

For help setting up your Go environment please follow the instructions for you platform at:

https://www.gopherguides.com/courses/preparing-your-environment-for-go-development

-> Node: Checking installation
✓ The `node` executable was found on your system at: /usr/local/bin/node

-> Node: Checking minimum version requirements
✓ Your version of Node, v10.15.3, meets the minimum requirements.

-> NPM: Checking installation
✓ The `npm` executable was found on your system at: /usr/local/bin/npm

-> NPM: Checking minimum version requirements
✓ Your version of NPM, 6.4.1, meets the minimum requirements.

-> Yarn: Checking installation
✓ The `yarnpkg` executable was found on your system at: /usr/local/bin/yarnpkg

-> Yarn: Checking minimum version requirements
✓ Your version of Yarn, 1.15.2, meets the minimum requirements.

-> PostgreSQL: Checking installation
✘ The `postgres` executable could not be found on your system.
For help setting up your Postgres environment please follow the instructions for you platform at:

https://www.postgresql.org/download/

-> MySQL: Checking installation
✘ The `mysql` executable could not be found on your system.
For help setting up your MySQL environment please follow the instructions for you platform at:

https://www.mysql.com/downloads/

-> SQLite3: Checking installation
✓ The `sqlite3` executable was found on your system at: /usr/bin/sqlite3

-> SQLite3: Checking minimum version requirements
✓ Your version of SQLite3, 3.24.0, meets the minimum requirements.

-> Cockroach: Checking installation
✘ The `cockroach` executable could not be found on your system.
For help setting up your Cockroach environment please follow the instructions for you platform at:

https://www.cockroachlabs.com/docs/stable/

-> Buffalo: Checking installation
✘ The `buffalo` executable could not be found on your system.
For help setting up your Buffalo environment please follow the instructions for you platform at:

https://gobuffalo.io/en/docs/installation

For help setting up your Go environment please follow the instructions for you platform at:

https://www.gopherguides.com/courses/preparing-your-environment-for-go-development

-> Buffalo: Application Details
Pwd         /Users/<SNIP>/go/src/<SNIP>
Root        /Users/<SNIP>/go/src/<SNIP>
GoPath      /Users/<SNIP>/go
PackagePkg  <SNIP>
ActionsPkg  <SNIP>/actions
ModelsPkg   <SNIP><SNIP>/models
GriftsPkg   <SNIP>/grifts
WithModules false
Name        gamoto
Bin         bin/gamoto
VCS         git
WithPop     true
WithSQLite  false
WithDep     false
WithWebpack true
WithNodeJs  true
WithYarn    true
WithDocker  true
WithGrifts  true
AsWeb       true
AsAPI       false
InApp       true
PackageJSON {map[]}

-> Buffalo: config/buffalo-app.toml
name = "gamoto"
bin = "bin/gamoto"
vcs = "git"
with_pop = true
with_sqlite = false
with_dep = false
with_webpack = true
with_nodejs = true
with_yarn = true
with_docker = true
with_grifts = true
as_web = true
as_api = false

-> Buffalo: config/buffalo-plugins.toml
[[plugin]]
  binary = "buffalo-plugins"
  go_get = "github.com/gobuffalo/buffalo-plugins"

[[plugin]]
  binary = "buffalo-pop"
  go_get = "github.com/gobuffalo/buffalo-pop"

[lukasschlueter]

-> Go: Checking PATH
✘ Your PATH (/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:~/Library/Python/2.7/bin) does not contain /Users/<CUT>/go/bin.

Without /Users/<CUT>/go/bin in your `PATH` any Go executables can not be run globally.

For help setting up your Go environment please follow the instructions for you platform at:

https://www.gopherguides.com/courses/preparing-your-environment-for-go-development

-> Buffalo: Checking installation
✘ The `buffalo` executable could not be found on your system.
For help setting up your Buffalo environment please follow the instructions for you platform at:

https://gobuffalo.io/en/docs/installation

For help setting up your Go environment please follow the instructions for you platform at:

https://www.gopherguides.com/courses/preparing-your-environment-for-go-development

This could be an issue with you path setup. Can you check that buffalo-plugins and buffalo-pop are in your path and try again?

[calston]

If a path setup causes this issue that’s a whole bug of its own

[lukasschlueter]

Buffalo is using plugins for different things, pop is one of those.
Please head to https://gobuffalo.io/en/docs/plugins for more information on this.
In theory, it should work without them being in $PATH, as long as $BUFFALO_PLUGIN_PATH or $GO_PATH are set.

As I could not reproduce this issue on my machine, I can't solve your problem without your help.

If a path setup causes this issue that’s a whole bug of its own

Well, this is a bug report after all, isn't it?

Also, it's specified as a buffalo requirement:

Before installing make sure you have the required dependencies installed:

• A working Go environment
• A configured $PATH environment variable that includes $GOPATH/bin.
• Go version 1.10.8 or greater.

[markbates]

I can't reproduce either, but I believe I know what happened.

You reported your second step as $ ~/go/bin/buffalo new test && cd test and then, correct me if I'm wrong, you edited the database.yml to change it to sqlite, correct? Nothing wrong with doing that, and with any other db you wouldn't have had trouble, but sqlite is a bit trickier.

The problem stems from the app not being generated initially with --db-type=sqlite3. Had you done that, the plugins system would've installed buffalo-pop for you with sqlite support baked in. Your config/buffalo-plugins.toml file would have looked like this:

[[plugin]]
  binary = "buffalo-plugins"
  go_get = "github.com/gobuffalo/buffalo-plugins"
  tags = ["sqlite"]

[[plugin]]
  binary = "buffalo-pop"
  go_get = "github.com/gobuffalo/buffalo-pop"
  tags = ["sqlite"]

The fix would be to either edit your config/buffalo-plugins.toml to include the sqlite tags.

Or you can add it with the buffalo plugins command:

$ buffalo plugins add --tags sqlite github.com/gobuffalo/buffalo-pop

Either should solve your problem. I think the real bug here is some missing documentation. Thanks for letting us know.

@lukasschlueter I'm transferring this to the gobuffalo/gobuffalo repo so documentation can be added to the site to talk about how you enable sqlite support after the fact.

[MaMrEzO]

Something wrong with it when running it different ways:
I added SQLite to the project after created it!

Here

go get -u -v -tags sqlite github.com/gobuffalo/pop/soda
go install -tags sqlite github.com/gobuffalo/pop/soda

#or may be rebuild buffalo itself with SQLite support

go get -u -v -tags sqlite github.com/gobuffalo/buffalo/buffalo

database.yml created with soda g config -t sqlite3

---
development:
  dialect: "sqlite3"
  database: /home/user/go/src/github.com/user/coke/sql_scripts/sqlite/coke.sqlite

test:
  dialect: "sqlite3"
  database: /home/user/go/src/github.com/user/coke/sql_scripts/sqlite/coke.sqlite

production:
  dialect: "sqlite3"
    database: /home/user/go/src/github.com/user/coke/sql_scripts/sqlite/coke.sqlite

buffalo-plugins.toml:

[[plugin]]
  binary = "buffalo-plugins"
  go_get = "github.com/gobuffalo/buffalo-plugins"
  tags = ["sqlite"]

[[plugin]]
  binary = "buffalo-pop"
  go_get = "github.com/gobuffalo/buffalo-pop"
  tags = ["sqlite"]

buffalo-app.toml:

name = "coke"
bin = "bin/coke"
vcs = "git"
with_pop = true
with_sqlite = true
with_dep = false
with_webpack = true
with_nodejs = true
with_yarn = true
with_docker = true
with_grifts = true
as_web = true
as_api = false

One line to log success and connection info (/models/models.go[func Init]):

func init() {
	var err error
	env := envy.Get("GO_ENV", "development")
	DB, err = pop.Connect(env)
	if err != nil {
		log.Fatal(err)
	}
	pop.Debug = env == "development"
	log.Println("Connection done: ", DB.String()) //<<<<<Report success connection
}

^^^^ ABOVE config are what I know about Buffalo and DB(pop/soda) if I missed sometihng correct me:

Afterward:

When running it with buffalo dev it seems works:

buffalo: 2019/12/27 20:58:01 === Building Completed (PID: 11243) (Time: 6.830132422s) ===
buffalo: 2019/12/27 20:58:01 === Running: tmp/coke-build (PID: 11356) ===
2019/12/27 20:58:03 Connection done:  /home/user/go/src/github.com/user/coke/sql_scripts/sqlite/coke.sqlite?_busy_timeout=5000&_fk=true
INFO[2019-12-27T20:58:04+03:30] Starting application at http://127.0.0.1:3000
INFO[2019-12-27T20:58:04+03:30] Starting Simple Background Worker

BUT RUNNING IT WITH DEBUGGING MODE
(GoLand| Run kind: Package|Package path: github.com/user/coke| Working directory:/home/user/go/src/github.com/user/coke)

will behave differently:

GOROOT=/USER_SPACE/Dev-Platform/go #gosetup
GOPATH=/home/user/go #gosetup
/USER_SPACE/Dev-Platform/go/bin/go build -o /tmp/___Debug___WebPack -gcflags "all=-N -l" github.com/user/coke #gosetup
/USER_SPACE/Applications/GoLand.2019.3/plugins/go/lib/dlv/linux/dlv --listen=localhost:35503 --headless=true --api-version=2 --check-go-version=false exec /tmp/___Debug___WebPack -- #gosetup
API server listening at: 127.0.0.1:35503
[POP] 2019/12/27 21:01:19 warn - unable to load connection development: unsupported dialect 'sqlite3'
[POP] 2019/12/27 21:01:19 warn - unable to load connection test: unsupported dialect 'sqlite3'
[POP] 2019/12/27 21:01:19 warn - unable to load connection production: unsupported dialect 'sqlite3'
2019/12/27 21:01:22 could not find connection named development

Debugger finished with exit code 0

ALSO

$ buffalo pop -d
v4.13.0

[POP] 2019/12/27 20:55:44 debug - Loading config file from /home/user/go/src/github.com/user/coke/database.yml
[POP] 2019/12/27 20:55:44 warn - unable to load connection test: unsupported dialect 'sqlite3'
[POP] 2019/12/27 20:55:44 warn - unable to load connection production: unsupported dialect 'sqlite3'
[POP] 2019/12/27 20:55:44 warn - unable to load connection development: unsupported dialect 'sqlite3'

[MaMrEzO]

As sugested here:

$ buffalo new newone --db-type=sqlite
Usage:
  buffalo new [name] [flags]

Flags:
      --api                  skip all front-end code and configure for an API server
      --ci-provider string   specify the type of ci file you would like buffalo to generate [none, travis, gitlab-ci] (default "none")
      --config string        config file (default is $HOME/.buffalo.yaml)
      --db-type string       specify the type of database you want to use [cockroach, mysql, postgres] (default "postgres")
      --docker string        specify the type of Docker file to generate [none, multi, standard] (default "multi")
  -d, --dry-run              dry run
  -f, --force                delete and remake if the app already exists
  -h, --help                 help for new
      --module string        specify the root module (package) name. [defaults to 'automatic']
      --skip-config          skips using the config file
      --skip-pop             skips adding pop/soda to your app
      --skip-webpack         skips adding Webpack to your app
      --skip-yarn            use npm instead of yarn for frontend dependencies management
      --vcs string           specify the Version control system you would like to use [none, git, bzr] (default "git")
  -v, --verbose              verbosely print out the go get commands

ERRO[0000] Error: unknown dialect "sqlite" expecting one of cockroach, mysql, postgres 


$ buffalo new newone --db-type=sqlite3
Usage:
  buffalo new [name] [flags]

Flags:
      --api                  skip all front-end code and configure for an API server
      --ci-provider string   specify the type of ci file you would like buffalo to generate [none, travis, gitlab-ci] (default "none")
      --config string        config file (default is $HOME/.buffalo.yaml)
      --db-type string       specify the type of database you want to use [cockroach, mysql, postgres] (default "postgres")
      --docker string        specify the type of Docker file to generate [none, multi, standard] (default "multi")
  -d, --dry-run              dry run
  -f, --force                delete and remake if the app already exists
  -h, --help                 help for new
      --module string        specify the root module (package) name. [defaults to 'automatic']
      --skip-config          skips using the config file
      --skip-pop             skips adding pop/soda to your app
      --skip-webpack         skips adding Webpack to your app
      --skip-yarn            use npm instead of yarn for frontend dependencies management
      --vcs string           specify the Version control system you would like to use [none, git, bzr] (default "git")
  -v, --verbose              verbosely print out the go get commands

ERRO[0000] Error: unknown dialect "sqlite3" expecting one of cockroach, mysql, postgres 

[pplavetzki]

I also have the same problem when trying to run any tests using sqlite:

go test ./... or running test from VS Code

FAIL    github.com/pplavetzki/coke/models        0.008s
[POP] 2021/06/07 13:04:21 warn - unable to load connection production: unsupported dialect 'sqlite3'
[POP] 2021/06/07 13:04:21 warn - unable to load connection development: unsupported dialect 'sqlite3'
[POP] 2021/06/07 13:04:21 warn - unable to load connection test: unsupported dialect 'sqlite3'
2021/06/07 13:04:21 could not find connection named development

Running in buffalo dev produces no errors and connects just fine to sqlite3.

database.yml:

---
development:
  dialect: "sqlite3"
  database: /home/pplavetzki/go/src/github.com/pplavetzki/coke/sqlite/coke_development.sqlite

test:
  dialect: "sqlite3"
  database: /home/pplavetzki/go/src/github.com/pplavetzki/coke/sqlite/coke_test.sqlite

production:
  dialect: "sqlite3"
  database: /home/pplavetzki/go/src/github.com/pplavetzki/coke/sqlite/coke_production.sqlite

buffalo-plugins.toml:

[[plugin]]
  binary = "buffalo-pop"
  go_get = "github.com/gobuffalo/buffalo-pop/v2"
  tags = ["sqlite"]

buffalo-app.toml:

name = "coke"
bin = "bin/coke"
vcs = "git"
with_pop = true
with_sqlite = true
with_dep = false
with_webpack = true
with_nodejs = true
with_yarn = true
with_docker = true
with_grifts = true
as_web = true
as_api = false

Would love to know what the trick is to be able to run tests or debug without failing.