fix: classifier last token sometimes fail to capture the right values

[Will-ShaoHua]

Problem

Naive tokenization string.split(' ') will not work for strings like line=str, public void main( because there is no empty space. We need more sophisticate tokenization.

Solution

1. First use naive space characters to do the first filter and we only care about last token
2. Run regex to precisely capture real last token in the outcome of step 1

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.68%. Comparing base (2292bd7) to head (98a58c2).

Files with missing lines	Patch %	Lines
...age-server/inline-completion/utils/triggerUtils.ts	78.57%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2434      +/-   ##
==========================================
+ Coverage   62.66%   62.68%   +0.02%     
==========================================
  Files         266      266              
  Lines       59657    59671      +14     
  Branches     3832     3837       +5     
==========================================
+ Hits        37386    37407      +21     
+ Misses      22196    22189       -7     
  Partials       75       75              

Flag	Coverage Δ
unittests	62.68% <83.33%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.