[Java] Custom Metrics E2E Test

[Miqueasher]

This PR adds custom metrics to the existing sample app to verify public documentation is correct and functionality continues to work as changes happen in the future (Regression testing). Specifically for OpenTelemetry metrics using the CloudWatch Agent.

The changes made in this PR are:

1. 2 New imports to FrontendServiceController.java and dependencies to build.gradle.kts (sample app's frontend service for the 'main' instance and the dependency installer)
2. Global meter, and gauge/histogram/counter added to FrontendServiceController.java from OTEL SDK Metrics page
3. Update to environment variables in main.tf also added ec2 instance connect line for easier future debugging.
4. Update to jobs & env sections in java-ec2-default-test
5. Update to Cloudwatch agent config to include otlp ports
6. New Validator file to call updated predefined template file
7. Update to PredefinedExpectedTemplate to include new mustache file
8. New mustache file for validation of custom metrics asg & ec2-default OTEL_RESOURCE_ATTRIBUTES value and all related validation templates were swapped due to metric dimension limit being exceeded w/o the swap.

Git revert back to last passing CSHA: 82e7075

<Can we safely revert this commit if needed? If not, detail what must be done to safely revert and why it is needed.>

Ensure you've run the following tests on your changes and include the link below:

To do so, create a test.yml file with name: Test and workflow description to test your changes, then remove the file for your PR. Link your test run in your PR description. This process is a short term solution while we work on creating a staging environment for testing.

EC2-DEFAULT
EC2-ASG
EC2-ADOT-SIGV4

EKS
ECS
K8s

NOTE: TESTS RUNNING ON A SINGLE EKS CLUSTER CANNOT BE RUN IN PARALLEL. See the needs keyword to run tests in succession.

• Run Java EKS on e2e-playground in us-east-1 and eu-central-2
• Run Python EKS on e2e-playground in us-east-1 and eu-central-2
• Run metric limiter on EKS cluster e2e-playground in us-east-1 and eu-central-2
• Run EC2 tests in all regions
• Run K8s on a separate K8s cluster (check IAD test account for master node endpoints; these will change as we create and destroy clusters for OS patching)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[thpierce]

Why is there no change to the following files?

• .github/workflows/java-ec2-default-test.yml
• terraform/python/ec2/default/README.md