Results change with different batch size

[Simaex]

I try to use the code for implementing sentence embeddings and hit a strange issue. I've got different embeddings when changing batch size. When I run the code with suggested batch size 32, smaller numbers and up to 35 I receive one type of results that is not the same as when using TF.hub without Keras wrapper. But when I use batch size 36 or bigger I've got different type of results that is just the same as TF.hub without Keras wrapper. Please advise how to find source of my problem.
`# Function to build model
def build_model():
input_text = layers.Input(shape=(1,), dtype="string")
embedding = ElmoEmbeddingLayer()(input_text)
#dense = layers.Dense(128, activation='relu')(embedding)
#pred = layers.Dense(6, activation='softmax')(dense)

model = Model(inputs=[input_text], outputs=embedding)

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()

return model`