Filter on which files to save as assets

[ghostsquad]

Is there a way to filter which files within a directory will get save as assets?

[dmitshur]

Yes, by using an http.FileSystem wrapper that filters files.

You can/should use https://godoc.org/github.com/shurcooL/httpfs/filter.

The godoc includes some usage examples. You can also see here for an example of it being used in a real project.

[ghostsquad]

@shurcooL thanks for the super quick reply!! I'll take a look. Might be useful to add some more usage scenarios to the readme. I can submit a PR for this probably next week as I am still fairly new to Golang, and definitely new to this tool. 😃

[dmitshur]

Might be useful to add some more usage scenarios to the readme.

I agree. That's why I added documentation label to this issue. Let's use it to track that task.

Thanks for offering to send a PR! However, I'd prefer to take this myself. I've been wanting to overhaul the README, and mentioning filter package will be a part of that. So it'll be easier for me to do it as part of that than to review a PR.

[ghostsquad]

I'm still having a hard time putting the pieces together. First, I just want to be able to run a command that will generate a .go file as described in the readme, (that I can check into source control).

Later, once I understand go generate more, I'll want to incorporate that piece as well, but for now, trying to keep this as simple as possible.

Here's the situation

├── commands
│   ├── build
│   │   ├── main.go
│   │   └── thing-build.1.ronn.md
│   └── deploy
│       ├── main.go
│       └── thing-deploy.1.ronn.md
├── ronnstrings_generate.go

ronnstrings_generate.go

// +build ignore

package main

import (
	"log"
	"github.com/shurcooL/vfsgen"
	"go/build"
	"net/http"
	"github.com/shurcooL/httpfs/filter"
)

func importPathToDir(importPath string) string {
	p, err := build.Import(importPath, "", build.FindOnly)
	if err != nil {
		log.Fatalln(err)
	}
	return p.Dir
}

func main() {
	var FS = filter.Keep(
		http.Dir(importPathToDir("github.com/myproject/commands")),
		filter.FilesWithExtensions(".ronn.md"),
	)

	err := vfsgen.Generate(FS, vfsgen.Options{
		PackageName:  "data",
		VariableName: "RonnStrings",
	})
	if err != nil {
		log.Fatalln(err)
	}
}

result

$ go run ronnstrings_generate.go
writing ronnstrings_vfsdata.go
2017/12/22 18:05:36 can't stat file "/": open /: file does not exist

What am I doing wrong? Hopefully if I can get this going, you'll have enough UX information for a good example using filters.

[dmitshur]

@ghostsquad You're doing everything right, but unfortunately, you're running into the one known issue in the intersection of filter.Keep and filter.FilesWithExtensions. Please see shurcooL/httpfs#5.

For now, I'd suggest either using filter.FilesWithExtensions with filter.Skip only, or writing your custom keep function (e.g., similar to https://github.com/gopherjs/gopherjs/blob/444abdf/compiler/natives/fs.go#L27).

[ghostsquad]

Ah, simple fix then:

var FS = filter.Keep(
	http.Dir(importPathToDir("github.com/myproject/commands")),
	func(path string, fi os.FileInfo) bool {
		return fi.IsDir() || filter.FilesWithExtensions(".ronn.md")(path, fi)
	},
)

[ghostsquad]

it seems that this filter is pretty strict about what it considers an "extension". The following doesn't work:

• .ronn.md
• ronn.md

But md does.
I used this instead:

return fi.IsDir() || strings.HasSuffix(path,".ronn.md")

[ghostsquad]

The readme mentions the ability to do pre-processing. How do I pre-process the contents of a file before having that saved as an asset?

I can put this in a separate issue if that makes more sense.

Btw, I'm loving the simplicity of this. :)

[dmitshur]

Btw, I'm loving the simplicity of this. :)

Really glad to hear that! I really appreciate the simplicity too. :)

The readme mentions the ability to do pre-processing. How do I pre-process the contents of a file before having that saved as an asset?

Fair point, thanks for mentioning that. I could elaborate and point to some real examples of that when I re-work the README. No need for separate issue, this one is sufficient.

I will give an example now. In https://github.com/shurcooL/home/blob/080875be3b23bef30ca0606e19cc19a62476826e/assets/assets.go#L18, I use gopherjs_http.NewFS. If you read its documentation, it states:

NewFS returns an http.FileSystem that is exactly like source, except all Go packages are compiled to JavaScript with GopherJS.

For example:

/mypkg/foo.go
/mypkg/bar.go

Become replaced with:

/mypkg/mypkg.js

Where mypkg.js is the result of building mypkg with GopherJS.

Another example (that doesn't exist, just hypothetical) could be a filesystem that minifies all .css and .html files.

However, lately I've found myself relying less on such sweeping pre-processing facilities in favor of more explicit, specific actions on individual files or packages. These days I use gopherjs_http.Package more often, which works on a single Go package at a time.

[ghostsquad]

This is a little insane, lol:

// +build ignore

package main

import (
	"log"
	"github.com/shurcooL/vfsgen"
	"go/build"
	"net/http"
	"os"
	"strings"
	"github.com/ghostsquad/ronn2docopt"
	"bytes"
	"time"
	"fmt"
	"io"
	pathpkg "path"
)

func NewFS(source http.FileSystem, baseDir string) http.FileSystem {
	return &ronn2DocoptJS{
		source: source,
		baseDir: baseDir,
	}
}

type ronn2DocoptJS struct {
	source http.FileSystem
	baseDir string
}

type file struct {
	name    string
	modTime time.Time
	size    int64
	*bytes.Reader
}

func (f *file) Readdir(count int) ([]os.FileInfo, error) {
	return nil, fmt.Errorf("cannot Readdir from file %s", f.name)
}
func (f *file) Stat() (os.FileInfo, error) { return f, nil }

func (f *file) Name() string       { return f.name }
func (f *file) Size() int64        { return f.size }
func (f *file) Mode() os.FileMode  { return 0444 }
func (f *file) ModTime() time.Time { return f.modTime }
func (f *file) IsDir() bool        { return false }
func (f *file) Sys() interface{}   { return nil }

func (f *file) Close() error {
	return nil
}

type dir struct {
	name    string
	modTime time.Time
	entries []os.FileInfo
	pos     int // Position within entries for Seek and Readdir.
}

func (d *dir) Read([]byte) (int, error) {
	return 0, fmt.Errorf("cannot Read from directory %s", d.name)
}
func (d *dir) Close() error               { return nil }
func (d *dir) Stat() (os.FileInfo, error) { return d, nil }

func (d *dir) Name() string       { return d.name }
func (d *dir) Size() int64        { return 0 }
func (d *dir) Mode() os.FileMode  { return 0755 | os.ModeDir }
func (d *dir) ModTime() time.Time { return d.modTime }
func (d *dir) IsDir() bool        { return true }
func (d *dir) Sys() interface{}   { return nil }

func (d *dir) Seek(offset int64, whence int) (int64, error) {
	if offset == 0 && whence == io.SeekStart {
		d.pos = 0
		return 0, nil
	}
	return 0, fmt.Errorf("unsupported Seek in directory %s", d.name)
}

func (d *dir) Readdir(count int) ([]os.FileInfo, error) {
	if d.pos >= len(d.entries) && count > 0 {
		return nil, io.EOF
	}
	if count <= 0 || count > len(d.entries)-d.pos {
		count = len(d.entries) - d.pos
	}
	e := d.entries[d.pos : d.pos+count]
	d.pos += count
	return e, nil
}

func importPathToDir(importPath string) string {
	p, err := build.Import(importPath, "", build.FindOnly)
	if err != nil {
		log.Fatalln(err)
	}
	return p.Dir
}

func isRonnMd(name string) bool {
	return strings.HasSuffix(name, ".ronn.md")
}

func (fs *ronn2DocoptJS) Open(path string) (http.File, error) {
	f, err := fs.source.Open(path)
	if err != nil {
		return nil, err
	}
	defer f.Close()

	fi, err := f.Stat()
	if err != nil {
		return nil, err
	}

	if fi.IsDir() {
		fis, err := f.Readdir(0)
		if err != nil {
			return nil, err
		}

		var entries []os.FileInfo
		for _, fi := range fis {
			switch {
			case !fi.IsDir() && isRonnMd(fi.Name()):
				fallthrough
			case fi.IsDir():
				entries = append(entries, fi)
			}
		}

		return &dir{
			name:    fi.Name(),
			entries: entries,
			modTime: fi.ModTime(),
		}, nil
	}

	if isRonnMd(fi.Name()) {
		realPath := pathpkg.Join(fs.baseDir, path)
		fmt.Println(realPath)

+               // ***** PRE-PROCESS THE FILE HERE
		content, err := ronn2docopt.ConvertRonnFile(realPath)

		if err != nil {
			return nil, err
		}

		return &file{
			name:    fi.Name(),
			size:    int64(len(content)),
			modTime: time.Now(),
			Reader:  bytes.NewReader([]byte(content)),
		}, nil
	}

	return nil, &os.PathError{Op: "open", Path: path, Err: os.ErrNotExist}
}

func main() {
	projectDir := importPathToDir("github.com/myproject/commands")
	var FS = NewFS(http.Dir(projectDir), projectDir)

	err := vfsgen.Generate(FS, vfsgen.Options{
		PackageName:  "helpdata",
		VariableName: "RonnStrings",
		BuildTags:    "!dev",
		Filename: "helpdata/ronnstrings_vfsdata.go",
	})
	if err != nil {
		log.Fatalln(err)
	}
}

There's a lot to implement in order to provide pre-processing. I wouldn't wish this on anyone, but it was a super useful exercise for me.

[ghostsquad]

I've also noticed that directories that are skipped still show up in assets. If I change my code above to start at the root of my project, and then add some other filters:

if fi.IsDir() && (strings.HasPrefix(path, "/commands") || path == "/")  {

First I see this in the generate output:

2017/12/24 10:06:57 can't stat file "/.git": open /.git: file does not exist

which causes some alarm, and if I look at the vfsdata.go file, I see this, just the directory is saved.

fs["/.git"].(os.FileInfo),

I feel like as it walks a directory, if it encounters specifically a "Skip directory" error (and maybe a "Not exists" error), it should ignore that directory/file entirely, print a friendlier message.

it's a little bit confusing. I see the code path going back and forth between:
https://github.com/shurcooL/vfsgen/blob/master/generator.go#L90 and
https://github.com/shurcooL/httpfs/blob/master/vfsutil/walk.go, and especially since walkFn actually takes an error as a parameter. This seems to be where we are getting the log line in the terminal from. I'm still trying to find why the directory is still being saved as an asset.

This isn't specifically a blocker. I've replaced the error with filepath.SkipDir so that the error is less alarming. Though, I dislike how this file changes because skipped directories are still saved.

[ghostsquad]

I'll close this issue, as with a bit more research in making my own http filesystem, I've solved this problem.

Also, the immediate above problem with getting a can't stat error is resolved too. I think I had some code in the Readdir or Stat directory functions as I thought that was required to filter properly.