Data pump after production deploy on AWS Fargate

Get Help
#1

Hello everyone VSF Gurus!

These days I am trying to release Vue Storefront on AWS Fargate, one day I would also like to contribute to the documentation illustrating how I did it.

In summary, I created 5 docker images to mount on my services:

  • Nginx for reverse proxing
  • Phpfpm for Magento 2.4
  • Node JS for Vue Storefront frontend
  • Node JS for Vue Storefront API
  • Node JS for Mage2Vuestorefront

To these are added the AWS services for Elasticsearch and Redis (Elasticache).

I managed to get almost to a concrete result with production methods, but now I have some doubts.

Usually, in dev mode I followed these steps in this order to import the data:

  1. On the VSF API container: yarn db new
  2. On the Mage2vuestorefront container: node cli.js taxrule; node cli.js attributes; node cli.js categories; node cli.js products; node cli.js productcategories
  3. On the VSF API container: yarn db rebuild

At this point, can you confirm that the steps are correct?
Other than that, what exactly should happen during deployment?

Do you have to run these commands every time at each release, or just some of them? They are correct?

If so, I should run the db yarn new command at each deploy, have a cron that executes the commands in step 2 for mage2vuestorefront and finally another cron that runs the yarn db rebuild command on the api container, but it doesn’t seem the way cleaner.

Thanks so much!
Michael

#2

Hello,
The thing is - mage2vuestorefront is a deprecated thing. You should use magento2-vsbridge-indexer. With that, you will need to connect Magento directly to the Elasticsearch. After that, these 3 steps you mentioned will be redundant.

#3

Hi Fifciuu,
is this module compatible with Elasticsearch 7+ and Magento 2.4.*?
Can you confirm that the right version to be installed via composer is 2.0.1 and not 1.25.2?

Thank you,
Michael

#4

ES7 - Yes
Magento 2.4 - Hard to say, last version we’ve used it with is 2.3
2.0.1 or 1.25.2 - 2.0.1

#5

Thanks Fifciuu,
I was able to install and roughly configure the module.

I don’t understand some things though:

  1. When I used mage2vuestorefront locally, for example the “test” product was reachable once it was imported from localdomain.com/test.html now the url construction looks like this: stagingdomain.com/p/skutest1/test-1 - is it possible to set the seo friendly url again? I have seen that there are some configurations about it, but by making several attempts I could not fix it.

  2. Locally when I imported via mage2vuestorefront, if I accessed the redis docker container, I was able to see that it had written some keys. Now this doesn’t seem to happen anymore, can you tell me exactly when the keys on redis are written?

  3. I am testing all this using an elasticache (redis) t3.small machine but sometimes when I reindex too close, the service goes down. Can you advise me how to configure the module so that the data pump is less invasive?

I share here the current configurations of the module:

screencapture
screencapture2880×11558 2.11 MB

Thank you very much in advance, thanks to your support I am close to having an environment for staging and production of the project. I would love to do a full guide with this AWS Fargate setup once I get it all right.

Mike

#6
  1. You have a few settings for that in Plugin’s settings - last section (Use URL Keys - true, and one below to false). Then you want to enable config.seo.useUrlDispatcher in your pwa’s config. Products should be accessible via /<product_url_key>
  2. When you enabled cache in the config (useOutputCacheTagging & useOutputCache) and accessing some page/api catalog endpoint. Then it is stored in the Redis and served from the Redis next time.
  3. Not sure about that. Maybe you could make smaller batch size?
#7

Hi Fifciuu,

  1. I set it as you suggested, so with ‘xxxx’ => Yes, while the config.seo.useUrlDispatcher configuration on Vue Storefront (frontend) was already true. I reindexed from the plugin of the module of all the indexes in random order but I still have the url like this: / p / skutest1 / test-1 instead of /test.html as defined on Magento in Catalog> Products :: Search Engine Optimization Url :: URL Key = test

  2. Unfortunately if I enable those configurations both on the Vue Storefront Frontend configuration file and on the Vue Storefront API configuration file, the page continues to load indefinitely without ever giving an answer, do you have any ideas?

  3. Thanks for the tip, I have currently set up a larger (medium) instance and it seems to work fine.

Can you help me for these 3 points? The most urgent is definitely point 2. Excuse me for the constant requests but unfortunately I have an imminent deadline.

Thank you very much,
Mike

#8

UPDATE:

  1. It would seem that the friendly url problem is solved, maybe I forgot to clear the configuration cache on the Magento side.

2. For this problem I still have no idea how to solve, tomorrow I will try to do other tests to understand what is wrong.

  1. I confirm that the t3.medium instance seems to be fine, perhaps a bit large for a staging environment anyway, I will try to understand in the next few days.

New issue:
4. Maybe I missed some steps, but I have a problem with the Magento 2 module mentioned above. For example, if I create a new product and run the indexes, it would appear that a new product index is written on Elasticsearch, but the alias is not updated, still showing the old version on the frontend. I think this step, if I’m not mistaken, was solved by the yarn db rebuild command on the API application. Should it be automatic from the module? Or is there actually a step missing?

From what I understand, an index “vue_storefront_catalog_1_product_xxxxxxxx” is created which acts as an alias to the index “vue_storefront_catalog_1_product” which contains the data with the latest updates, while the data contained in the index which aliases the index “vue_storefront_catalog_product” remains the same, with outdated data. In fact, if I execute cURLs with the REMOVE and ADD methods in order to reassociate the index to the correct alias, I can finally see the updated frontend. Maybe there is some configuration error on the applications?

Thanks again,
Michael

#9

2.What do you have in the terminal with the server? Maybe you did not configure connection to Redis properly that’s why it infinitely tries to connect to it without success?

4.I am pretty sure it should work fine. Have you checked logs of the Magento? Maybe there are some hints

#10

Hi Fifciuu,
is VSF compatible with Redis 6.x? I am using this version on Elasticache.

As for the alias problem, tomorrow I’ll investigate it better, can you tell me in the meantime if VSF is compatible with Elasticsearch 7.9.x?

Thanks,
Michael

#11

I’ve just talked with our devops about that.

For ES, the newest one we’ve deployed is 7.8 - but you could try 7.9. Probably, it will work.
About Redis, default version is 4.

#12

Thanks Fifciuu,
I’ve solved the problem of the ES aliases, my fault, I had missed a piece of the module’s documentation that instructed to do this:

image
image1758×1012 74.5 KB

So by adding the suffix _1 to the fields elasticsearch.index (VSF), elasticsearch.indices (VSF API) and having for now only a storeview with ID 1, everything is back to working.

Tomorrow or next week I will make other attempts on redis, although I will probably not be able to use it for my development.

Thanks,
Michael