Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v2]Added Auto Index Rollover #6500

Closed
wants to merge 10 commits into from
Closed

[v2]Added Auto Index Rollover #6500

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ebe2111
first step
Manik2708 Jan 6, 2025
c58fd66
e2e
Manik2708 Jan 7, 2025
3d5abf5
e2e
Manik2708 Jan 7, 2025
48747f4
finishing step
Manik2708 Jan 7, 2025
35366b2
finishing step
Manik2708 Jan 7, 2025
a8d7908
Merge branch 'main' into es-roll
Manik2708 Jan 7, 2025
2e42c86
finishing step
Manik2708 Jan 7, 2025
97c4020
finishing step
Manik2708 Jan 7, 2025
12aed54
cron job
Manik2708 Jan 10, 2025
dd85780
cron job test removed
Manik2708 Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions cmd/es-rollover/app/actions.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import (
"github.com/jaegertracing/jaeger/pkg/es/client"
)

func newESClient(endpoint string, cfg *Config, tlsCfg *tls.Config) client.Client {
func NewESClient(endpoint string, cfg *Config, tlsCfg *tls.Config) client.Client {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why can we not use ES client from pkg/es?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So are you saying to use the olivere client instead of those used in cmd/es-rollover?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not olivere client, but our internal abstraction from pkg/es

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not olivere client, but our internal abstraction from pkg/es

But then should I copy this function in factory?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest you step away from the main issue for now and think about if we need to do a bit of refactoring of the existing code first. I see no reason why all the cmd/es-*** would have to be creating new HTTP client and doing some bespoke stuff that is completely independent of how es storage implementation is written. Everything should be going through pkg/es abstraction.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok then, I could think of these steps on refactoring:

  1. Refactoring the configs of rollover and lookback, encorporating them into the single common config of es-rollover. This will be very helpful while integrating the rollover with jaeger binary.
  2. As said by you changing the http clients to our abstraction (pkg/es).
  3. Changing the Unit and UnitCount to a single parameter time.Duration.
    I am thinking of seperate PRs for each. For 1st point, if this suggestion looks good to you, I will change accordingly in the PR [refactor] Seperated the parts of ES-Rollover Config common with ES Config #6516

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes separate PRs sounds good

httpClient := &http.Client{
Timeout: time.Duration(cfg.Timeout) * time.Second,
Transport: &http.Transport{
Expand Down Expand Up @@ -57,7 +57,7 @@ func ExecuteAction(opts ActionExecuteOptions, createAction ActionCreatorFunction
return fmt.Errorf("TLS configuration failed: %w", err)
}

esClient := newESClient(opts.Args[0], &cfg, tlsCfg)
esClient := NewESClient(opts.Args[0], &cfg, tlsCfg)
action := createAction(esClient, cfg)
return action.Do()
}
49 changes: 18 additions & 31 deletions cmd/es-rollover/app/flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,65 +10,52 @@ import (
"go.opentelemetry.io/collector/config/configtls"

"github.com/jaegertracing/jaeger/pkg/config/tlscfg"
"github.com/jaegertracing/jaeger/pkg/es/config"
)

var tlsFlagsCfg = tlscfg.ClientFlagsConfig{Prefix: "es"}
var (
tlsFlagsCfg = tlscfg.ClientFlagsConfig{Prefix: "es"}
esrolloverCfg = config.EsRolloverFlagConfig{}
)

const (
indexPrefix = "index-prefix"
archive = "archive"
username = "es.username"
password = "es.password"
useILM = "es.use-ilm"
ilmPolicyName = "es.ilm-policy-name"
timeout = "timeout"
skipDependencies = "skip-dependencies"
adaptiveSampling = "adaptive-sampling"
indexPrefix = "index-prefix"
username = "es.username"
password = "es.password"
useILM = "es.use-ilm"
)

// Config holds the global configurations for the es rollover, common to all actions
type Config struct {
IndexPrefix string
Archive bool
Username string
Password string
TLSEnabled bool
ILMPolicyName string
UseILM bool
Timeout int
SkipDependencies bool
AdaptiveSampling bool
TLSConfig configtls.ClientConfig
config.RolloverOptions
IndexPrefix string
Username string
Password string
TLSEnabled bool
UseILM bool
TLSConfig configtls.ClientConfig
}

// AddFlags adds flags
func AddFlags(flags *flag.FlagSet) {
esrolloverCfg.AddFlagsForRolloverOptions(flags)
flags.String(indexPrefix, "", "Index prefix")
flags.Bool(archive, false, "Handle archive indices")
flags.String(username, "", "The username required by storage")
flags.String(password, "", "The password required by storage")
flags.Bool(useILM, false, "Use ILM to manage jaeger indices")
flags.String(ilmPolicyName, "jaeger-ilm-policy", "The name of the ILM policy to use if ILM is active")
flags.Int(timeout, 120, "Number of seconds to wait for master node response")
flags.Bool(skipDependencies, false, "Disable rollover for dependencies index")
flags.Bool(adaptiveSampling, false, "Enable rollover for adaptive sampling index")
tlsFlagsCfg.AddFlags(flags)
}

// InitFromViper initializes config from viper.Viper.
func (c *Config) InitFromViper(v *viper.Viper) {
c.RolloverOptions = esrolloverCfg.InitRolloverOptionsFromViper(v)
c.IndexPrefix = v.GetString(indexPrefix)
if c.IndexPrefix != "" {
c.IndexPrefix += "-"
}
c.Archive = v.GetBool(archive)
c.Username = v.GetString(username)
c.Password = v.GetString(password)
c.ILMPolicyName = v.GetString(ilmPolicyName)
c.UseILM = v.GetBool(useILM)
c.Timeout = v.GetInt(timeout)
c.SkipDependencies = v.GetBool(skipDependencies)
c.AdaptiveSampling = v.GetBool(adaptiveSampling)
tlsCfg, err := tlsFlagsCfg.InitFromViper(v)
if err != nil {
panic(err)
Expand Down
40 changes: 19 additions & 21 deletions cmd/es-rollover/app/init/action_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/jaegertracing/jaeger/cmd/es-rollover/app"
"github.com/jaegertracing/jaeger/pkg/es/client"
"github.com/jaegertracing/jaeger/pkg/es/client/mocks"
"github.com/jaegertracing/jaeger/pkg/es/config"
)

func TestIndexCreateIfNotExist(t *testing.T) {
Expand Down Expand Up @@ -90,8 +91,8 @@ func TestRolloverAction(t *testing.T) {
},
config: Config{
Config: app.Config{
Archive: true,
UseILM: true,
RolloverOptions: config.RolloverOptions{Archive: true},
UseILM: true,
},
},
expectedErr: errors.New("ILM is supported only for ES version 7+"),
Expand All @@ -104,8 +105,8 @@ func TestRolloverAction(t *testing.T) {
expectedErr: errors.New("version error"),
config: Config{
Config: app.Config{
Archive: true,
UseILM: true,
RolloverOptions: config.RolloverOptions{Archive: true},
UseILM: true,
},
},
},
Expand All @@ -118,9 +119,8 @@ func TestRolloverAction(t *testing.T) {
expectedErr: errors.New("ILM policy myilmpolicy doesn't exist in Elasticsearch. Please create it and re-run init"),
config: Config{
Config: app.Config{
Archive: true,
UseILM: true,
ILMPolicyName: "myilmpolicy",
RolloverOptions: config.RolloverOptions{Archive: true, ILMPolicyName: "myilmpolicy"},
UseILM: true,
},
},
},
Expand All @@ -133,9 +133,8 @@ func TestRolloverAction(t *testing.T) {
expectedErr: errors.New("error getting ilm policy"),
config: Config{
Config: app.Config{
Archive: true,
UseILM: true,
ILMPolicyName: "myilmpolicy",
RolloverOptions: config.RolloverOptions{Archive: true, ILMPolicyName: "myilmpolicy"},
UseILM: true,
},
},
},
Expand All @@ -148,8 +147,8 @@ func TestRolloverAction(t *testing.T) {
expectedErr: errors.New("error creating template"),
config: Config{
Config: app.Config{
Archive: true,
UseILM: false,
RolloverOptions: config.RolloverOptions{Archive: true},
UseILM: false,
},
},
},
Expand All @@ -164,8 +163,8 @@ func TestRolloverAction(t *testing.T) {
expectedErr: errors.New("error getting jaeger indices"),
config: Config{
Config: app.Config{
Archive: true,
UseILM: false,
RolloverOptions: config.RolloverOptions{Archive: true},
UseILM: false,
},
},
},
Expand All @@ -184,8 +183,8 @@ func TestRolloverAction(t *testing.T) {
expectedErr: errors.New("error creating aliases"),
config: Config{
Config: app.Config{
Archive: true,
UseILM: false,
RolloverOptions: config.RolloverOptions{Archive: true},
UseILM: false,
},
},
},
Expand All @@ -204,8 +203,8 @@ func TestRolloverAction(t *testing.T) {
expectedErr: nil,
config: Config{
Config: app.Config{
Archive: true,
UseILM: false,
RolloverOptions: config.RolloverOptions{Archive: true},
UseILM: false,
},
},
},
Expand All @@ -225,9 +224,8 @@ func TestRolloverAction(t *testing.T) {
expectedErr: nil,
config: Config{
Config: app.Config{
Archive: true,
UseILM: true,
ILMPolicyName: "jaeger-ilm",
RolloverOptions: config.RolloverOptions{Archive: true, ILMPolicyName: "jaeger-ilm"},
UseILM: true,
},
},
},
Expand Down
31 changes: 19 additions & 12 deletions cmd/es-rollover/app/lookback/action_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/jaegertracing/jaeger/cmd/es-rollover/app"
"github.com/jaegertracing/jaeger/pkg/es/client"
"github.com/jaegertracing/jaeger/pkg/es/client/mocks"
"github.com/jaegertracing/jaeger/pkg/es/config"
)

func TestLookBackAction(t *testing.T) {
Expand Down Expand Up @@ -81,11 +82,13 @@ func TestLookBackAction(t *testing.T) {
}).Return(nil)
},
config: Config{
Unit: "days",
UnitCount: 1,
LookBackOptions: config.LookBackOptions{
Unit: "days",
UnitCount: 1,
},
Config: app.Config{
Archive: true,
UseILM: true,
RolloverOptions: config.RolloverOptions{Archive: true},
UseILM: true,
},
},
expectedErr: nil,
Expand All @@ -96,11 +99,13 @@ func TestLookBackAction(t *testing.T) {
indexClient.On("GetJaegerIndices", "").Return(indices, errors.New("get indices error"))
},
config: Config{
Unit: "days",
UnitCount: 1,
LookBackOptions: config.LookBackOptions{
Unit: "days",
UnitCount: 1,
},
Config: app.Config{
Archive: true,
UseILM: true,
RolloverOptions: config.RolloverOptions{Archive: true},
UseILM: true,
},
},
expectedErr: errors.New("get indices error"),
Expand All @@ -111,11 +116,13 @@ func TestLookBackAction(t *testing.T) {
indexClient.On("GetJaegerIndices", "").Return([]client.Index{}, nil)
},
config: Config{
Unit: "days",
UnitCount: 1,
LookBackOptions: config.LookBackOptions{
Unit: "days",
UnitCount: 1,
},
Config: app.Config{
Archive: true,
UseILM: true,
RolloverOptions: config.RolloverOptions{Archive: true},
UseILM: true,
},
},
expectedErr: nil,
Expand Down
17 changes: 5 additions & 12 deletions cmd/es-rollover/app/lookback/flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,30 +9,23 @@ import (
"github.com/spf13/viper"

"github.com/jaegertracing/jaeger/cmd/es-rollover/app"
"github.com/jaegertracing/jaeger/pkg/es/config"
)

const (
unit = "unit"
unitCount = "unit-count"
defaultUnit = "days"
defaultUnitCount = 1
)
var esrolloverCfg = config.EsRolloverFlagConfig{}

// Config holds configuration for index cleaner binary.
type Config struct {
app.Config
Unit string
UnitCount int
config.LookBackOptions
}

// AddFlags adds flags for TLS to the FlagSet.
func (*Config) AddFlags(flags *flag.FlagSet) {
flags.String(unit, defaultUnit, "used with lookback to remove indices from read alias e.g, days, weeks, months, years")
flags.Int(unitCount, defaultUnitCount, "count of UNITs")
esrolloverCfg.AddFlagsForLookBackOptions(flags)
}

// InitFromViper initializes config from viper.Viper.
func (c *Config) InitFromViper(v *viper.Viper) {
c.Unit = v.GetString(unit)
c.UnitCount = v.GetInt(unitCount)
c.LookBackOptions = esrolloverCfg.InitLookBackFromViper(v)
}
5 changes: 3 additions & 2 deletions cmd/es-rollover/app/rollover/action_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"github.com/jaegertracing/jaeger/cmd/es-rollover/app"
"github.com/jaegertracing/jaeger/pkg/es/client"
"github.com/jaegertracing/jaeger/pkg/es/client/mocks"
"github.com/jaegertracing/jaeger/pkg/es/config"
)

func TestRolloverAction(t *testing.T) {
Expand Down Expand Up @@ -115,9 +116,9 @@ func TestRolloverAction(t *testing.T) {

rolloverAction := Action{
Config: Config{
Conditions: test.conditions,
RollBackOptions: config.RollBackOptions{Conditions: test.conditions},
Config: app.Config{
Archive: true,
RolloverOptions: config.RolloverOptions{Archive: true},
},
},
IndicesClient: indexClient,
Expand Down
12 changes: 5 additions & 7 deletions cmd/es-rollover/app/rollover/flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,23 @@ import (
"github.com/spf13/viper"

"github.com/jaegertracing/jaeger/cmd/es-rollover/app"
"github.com/jaegertracing/jaeger/pkg/es/config"
)

const (
conditions = "conditions"
defaultRollbackCondition = "{\"max_age\": \"2d\"}"
)
var esrolloverCfg = config.EsRolloverFlagConfig{}

// Config holds configuration for index cleaner binary.
type Config struct {
app.Config
Conditions string
config.RollBackOptions
}

// AddFlags adds flags for TLS to the FlagSet.
func (*Config) AddFlags(flags *flag.FlagSet) {
flags.String(conditions, defaultRollbackCondition, "conditions used to rollover to a new write index")
esrolloverCfg.AddFlagsForRollBackOptions(flags)
}

// InitFromViper initializes config from viper.Viper.
func (c *Config) InitFromViper(v *viper.Viper) {
c.Conditions = v.GetString(conditions)
c.RollBackOptions = esrolloverCfg.InitRollBackFromViper(v)
}
Loading
Loading