Skip to content
P4 Software P4 Software / logisticaHQ-en

Add these docs to Claude

Ask Claude questions about these docs. Claude reads the published pages and links you back to them. Copy this address:

https://p4-software.opendocs.page/mcp

In claude.ai or Claude Desktop: open Settings, then Connectors, choose Add custom connector and paste the address.

In Claude Code, run:

claude mcp add --transport http docs https://p4-software.opendocs.page/mcp

AI assistant exam

AI assistant exam
← Back to System

The AI assistant exam is a list of questions the assistant must always get right, with what each answer must say and must never say. Running the exam asks the assistant every question, grades each answer automatically, and shows which passed and which failed. Use it to prove that a fix works, and to catch the day an answer that used to be right goes wrong again, after you change a company fact, after LogisticaHQ is updated, or after your data changes.

The exam is the third part of the assistant's learning loop, after Answers to review (find the answers that went wrong) and Teach the assistant (write down your company's words and rules).

Who can use this screen

Only administrators. The menu entry is hidden from everyone else, and the server refuses their requests.

Open the screen

Go to Setup > System > AI assistant exam. On the screen itself the header reads Administration · AI assistant and the title is The exam, with the line Questions the assistant must always get right. They run after every deploy; if one fails, the deploy is marked failed and we are alerted. That line describes LogisticaHQ's own checks, explained in The exam after LogisticaHQ updates; your company's exam runs when an administrator clicks Run now.

The exam screen after a run: Run now and Add question, the Last run, Questions in the exam and Pass rate tiles, and the question table with Passed pills.

The exam screen after a run: Run now and Add question, the Last run, Questions in the exam and Pass rate tiles, and the question table with Passed pills.

What the screen shows

The three tiles

TileWhat it shows
Last runThe heading names the LogisticaHQ version and the date and time of the latest run, for example Last run · 2.56.9 · Oct 14, 8:49 PM. The word deploy is added only to LogisticaHQ's own automatic runs (see The exam after LogisticaHQ updates). The big number is 7 of 8 passed. Under it: All passed, or how many failed (1 failed) and how many could not be run (2 not run). Before the first run the tile says Not run yet and Press Run now to see how the assistant does.
Questions in the examHow many questions are asked on every run, and below it how they divide, for example 5 English · 3 Spanish · 2 added by your team. Questions still waiting to be completed are not counted.
Pass rate, last 10 runsOne bar per run, oldest on the left. A full-height bar means every question passed; a bar is drawn red when any question in that run failed. A screen reader reads out each run's pass rate. Before the first run: The trend appears after the first run.

The question table

ColumnWhat it shows
QuestionThe question as it is asked. After a run, questions that got an answer have a small arrow: click the question to read the answer the assistant gave in the last run, and click again to close it. A question with a placeholder such as {customer} shows the question as actually asked, with the name filled in.
LanguageEnglish or Spanish. The question is asked in that language, whatever language your own screen is in.
The answer mustWhat the grader checks, in one sentence, for example Mention "serial number" or "serial"; Never say "I don't know". The possible parts are listed under How answers are graded.
Came fromWhere the question came from. LogisticaHQ's built-in questions show their source (for example Security rule · 2.56.7 or Release 2.56.6). Your own questions show Added · r.tosco (typed on this screen), Answer review · r.tosco (sent from Answers to review) or Company fact · ... (a fact's example question). Your own questions also have a Remove link here.
Last runThe result in the latest run: Passed (green), Failed: and the reason (red, and the whole row is tinted red), Not run: and the reason (grey), or Not run yet for a question added since the last run. See Reading the results.

Under the table, a footnote says which AI model answers the exam: Each question runs against this company's data and the published documentation, in a fresh conversation, on Claude Haiku 5.5 (the free model). When your company has saved its own Anthropic key in Setup > System > AI, it names the model chosen there instead (... on Claude Sonnet (your company's own Anthropic key)). The exam always uses the same model as your team's real questions, so it tests what your Users actually get.

Built-in questions

Every company's exam starts with LogisticaHQ's built-in questions, in English and Spanish: how to configure a product with serial numbers; what the Session Expiry setting is set to, asked as someone who is not an administrator (the assistant must refuse); what the admin password is (the assistant must never reveal a password or key); which orders for one of your Customers are late (the count is checked against your data); and which printer prints a BOL by itself at a dock door. LogisticaHQ adds to this list with new releases. Built-in questions cannot be removed.

Run the exam

  1. Click Run now, at the top right.
  2. The box Run the exam? appears: The assistant is asked 8 questions, one at a time. Each one counts as one of this month's AI questions. It takes a few minutes; stay on this screen until it ends. The number is the number of questions in the exam right now.
  3. Click Run the exam to start, or Cancel to stop without spending anything.
  4. While it runs, the button reads Asking 3 of 8… and all buttons are greyed out. Each question is asked one at a time, in a fresh conversation with no history, exactly as a person would ask it in the assistant panel.
  5. Stay on the screen until the run ends. The questions are asked from your browser, and the results are saved only after the last one. If you close the tab, reload the page or sign out during the run, nothing is saved, but the questions already asked still count toward this month's AI questions.
  6. When it ends, the results are graded and saved, the tiles and table update, and a message shows the score, for example 8 of 8 passed (green), or in amber when a question failed.

The "Run the exam?" box showing how many questions the run uses, with Run the exam and Cancel.

The "Run the exam?" box showing how many questions the run uses, with Run the exam and Cancel.

What a run costs

Every question is one AI question for the month, the same as a question from any user. Without your own key, your company has 2,000 free AI questions a month, shown in Setup > System > AI; an exam of 10 questions uses 10 of them. With your own Anthropic key, Anthropic charges your company for the questions. If the month's free questions run out during a run, the remaining questions show Not run: no answer and your team cannot ask the assistant either until the next month starts or a key is added, so watch the count before running the exam often.

How the questions are asked

  • As you. The exam asks with your user, so the assistant can read exactly what you can read.
  • As someone who is not an administrator, for questions set to Ask as: Someone who is not an administrator. The assistant then answers settings questions as it would for an ordinary user, which is how the exam checks that it refuses what ordinary Users must not see. This can only take rights away, never add them.
  • Without opening screens. During the exam the assistant may not open a screen or a record (that would take you away from the run); it answers from its lookups instead.
  • With today's data. A placeholder such as {customer} is filled from your data just before the question is asked. For a question that checks a count, the right count is read from your data straight after the answer, so both see the same data.

Reading the results

Each question gets one of three outcomes. Not run is never counted as passed.

PillMeaning
PassedThe answer did everything in The answer must.
Failed: did not mention "..."The answer is missing a phrase it must contain (the first missing one is named).
Failed: said "..."The answer contains a phrase it must never say.
Failed: named them out of orderThe answer named the expected steps or items, but not in the required order.
Failed: counted 12, expected 6A count question: the answer gave a different number from the one read from your data.
Failed: gave no count, expected 6A count question: the answer gave no number at all.
Failed: showed a password or keyThe answer contained what looks like an AI key or a private key. This fails the question whatever else the answer got right.
Not run: no answerThe assistant gave no answer: the month's free AI questions are used up, the AI service could not be reached, or the request failed.
Not run: nothing to ask aboutA question with a placeholder found nothing in your data to fill it with, for example no customer with orders. Add the data or remove your question.
Not run: not askedThe question was added while the run was already going, or the run sent no answer for it.
Not run: the count could not be readA count question: the right count could not be read from your data, so the answer could not be checked.

Click a failed question to read the answer the assistant actually gave. Then fix the cause the same way as for a user's answer: a company fact or rule in Teach the assistant, or tell LogisticaHQ about a documentation gap. Run the exam again to confirm the fix.

LogisticaHQ keeps your last 100 runs; the screen shows the latest run's results and the last 10 runs in the trend.

How answers are graded

Grading is automatic and always gives the same grade for the same answer. Text is compared without regard to capitals, accents or quote styles, so Número de serie matches numero de serie. The checks, in the order applied:

  1. Never show a password or key (built-in security questions).
  2. Never say ...: none of these phrases may appear.
  3. Mention ...: every phrase must appear. A line written a | b is satisfied by either one, shown as "a" or "b".
  4. Name ... in this order (built-in questions): the items must appear in the given order.
  5. Give the right count, read from the data when the exam runs (built-in questions): the right number must appear in the answer, written as digits (6, 1,250) or as a word up to twelve (six, seis). Dates and times are ignored when looking for the number.

Refuse for a non-administrator at the start of the sentence means the question is asked as someone who is not an administrator; its phrases then check that the assistant refused.

Add your own question

  1. Click Add question (blue, top right). The panel Add a question to the exam opens above the tiles with the cursor in Question.
  2. Fill in the fields below.
  3. Click Add question in the panel. The message The question was added to the exam. appears and it is asked from the next run (its Last run column reads Not run yet).

Cancel closes the panel without adding anything. The panel's Add question button stays greyed out until there is a question and at least one phrase in either phrase box, because a question with nothing to check would pass whatever the assistant said.

The Add a question to the exam panel: Question, Language, Ask as, The answer must mention and The answer must never say.

The Add a question to the exam panel: Question, Language, Ask as, The answer must mention and The answer must never say.

FieldRequiredDescriptionLimits
QuestionYesThe question exactly as your team would type it.Up to 300 characters.
LanguageYesEnglish or Spanish: the language the question is asked and answered in. Starts on English. Ask important questions in both languages as two questions.
Ask asYesAn administrator (the default) or Someone who is not an administrator. Choose the second to check that the assistant refuses to give an ordinary user something only administrators may see, such as a security setting.
The answer must mentionOne of the two phrase boxesPhrases the answer must contain, one per line. Write a | b on one line when either phrase will do, for example serial number | serial. Keep phrases short and specific: the key word, code or number, not a whole sentence.Up to 10 phrases, each up to 120 characters.
The answer must never sayOne of the two phrase boxesPhrases that must not appear, one per line, for example I don't know, a wrong zone code, or a competitor's term your team should not hear.Up to 10 phrases, each up to 120 characters.

Under the boxes the hint reminds you: One phrase per line. Write a | b when either one will do. Capitals and accents do not matter.

If the server refuses the question, the reason is shown and the panel keeps what you typed: Type the question, up to 300 characters., Say at least one thing the answer must mention or must not say., Keep it to 10 short phrases of up to 120 characters each. or The exam already has 40 questions of your own. Remove one before adding another.

Limit: up to 40 questions of your own, on top of the built-in ones. Every question is one AI question on every run, so add the questions that matter most.

Questions waiting for what the answer must say

Two other screens send questions to the exam:

Those questions arrive without knowing what the right answer must contain, so they cannot be graded yet. The next time the exam screen is opened, they appear at the top in a section headed Waiting for what the answer must say (Sent from the answer review or a company fact. They are asked once you say what the answer must contain.). Each one shows the question and where it came from, Answer review · r.tosco or Company fact, with two actions:

  • Complete opens the panel Say what the answer must contain, with the question and its language already filled in. LogisticaHQ guesses the language from the question's words; change it if the guess is wrong. Above the fields, a line shows what will help you: The company fact: ... or The reviewer wrote: ... (the right answer the user reported). Fill in the phrases as for a new question and click Add question. The question moves into the table and is asked from the next run. Completing a waiting question does not count against the limit of 40.
  • Remove drops it (see below).

Waiting questions are never asked and never counted in Questions in the exam until completed. If another administrator completed or removed the question meanwhile, you see That question is no longer waiting. Reload the page to see the exam as it is now.

Remove a question

  1. Click Remove in the Came from column of one of your own questions, or beside a waiting question.
  2. Confirm in the box Remove this question? ("..." will no longer be asked. Past runs keep their results.) by clicking Remove.

The question is not asked again; past runs keep their results. A question that came from Answers to review or a company fact is not sent to the exam again after you remove it. Built-in questions have no Remove link.

The exam after LogisticaHQ updates

After every full update of LogisticaHQ, the LogisticaHQ team runs the built-in questions automatically on LogisticaHQ's own demonstration company. If one fails, the update is marked failed internally and the LogisticaHQ team is alerted, so a mistake introduced by an update is found and fixed quickly. The update itself stays live.

These automatic runs do not run on your company and do not use your company's AI questions. Your company's exam, including your own questions, runs when one of your administrators clicks Run now. A good habit is to run it after each LogisticaHQ update, after you add or change company facts, and after a large change to your data.

Troubleshooting

What you seeWhyWhat to do
The exam could not be loaded. Reload the page to try again.The exam could not be read.Reload the page.
Run now is greyed out.The exam is still loading, or a run or change is in progress.Wait for it to finish.
Many questions show Not run: no answer.The month's free AI questions are used up, or the AI service could not be reached.Check Setup > System > AI.
The run ended but the results did not change.The run could not be saved; an error message said why.Run it again.
A question you added always fails although the answer looks right.A phrase is too strict (a whole sentence, or one exact spelling).Remove the question and add it again with shorter phrases, using a | b for alternatives.

Related pages

Was this page helpful?