P4 Software
/
logisticaHQ-en
The AI assistant exam is a list of questions the assistant must always get right, with what each answer must say and must never say. Running the exam asks the assistant every question, grades each answer automatically, and shows which passed and which failed. Use it to prove that a fix works, and to catch the day an answer that used to be right goes wrong again, after you change a company fact, after LogisticaHQ is updated, or after your data changes.
The exam is the third part of the assistant's learning loop, after Answers to review (find the answers that went wrong) and Teach the assistant (write down your company's words and rules).
Only administrators. The menu entry is hidden from everyone else, and the server refuses their requests.
Go to Setup > System > AI assistant exam. On the screen itself the header reads Administration · AI assistant and the title is The exam, with the line Questions the assistant must always get right. They run after every deploy; if one fails, the deploy is marked failed and we are alerted. That line describes LogisticaHQ's own checks, explained in The exam after LogisticaHQ updates; your company's exam runs when an administrator clicks Run now.

The exam screen after a run: Run now and Add question, the Last run, Questions in the exam and Pass rate tiles, and the question table with Passed pills.
| Tile | What it shows |
| Last run | The heading names the LogisticaHQ version and the date and time of the latest run, for example Last run · 2.56.9 · Oct 14, 8:49 PM. The word deploy is added only to LogisticaHQ's own automatic runs (see The exam after LogisticaHQ updates). The big number is 7 of 8 passed. Under it: All passed, or how many failed (1 failed) and how many could not be run (2 not run). Before the first run the tile says Not run yet and Press Run now to see how the assistant does. |
| Questions in the exam | How many questions are asked on every run, and below it how they divide, for example 5 English · 3 Spanish · 2 added by your team. Questions still waiting to be completed are not counted. |
| Pass rate, last 10 runs | One bar per run, oldest on the left. A full-height bar means every question passed; a bar is drawn red when any question in that run failed. A screen reader reads out each run's pass rate. Before the first run: The trend appears after the first run. |
| Column | What it shows |
| Question | The question as it is asked. After a run, questions that got an answer have a small arrow: click the question to read the answer the assistant gave in the last run, and click again to close it. A question with a placeholder such as {customer} shows the question as actually asked, with the name filled in. |
| Language | English or Spanish. The question is asked in that language, whatever language your own screen is in. |
| The answer must | What the grader checks, in one sentence, for example Mention "serial number" or "serial"; Never say "I don't know". The possible parts are listed under How answers are graded. |
| Came from | Where the question came from. LogisticaHQ's built-in questions show their source (for example Security rule · 2.56.7 or Release 2.56.6). Your own questions show Added · r.tosco (typed on this screen), Answer review · r.tosco (sent from Answers to review) or Company fact · ... (a fact's example question). Your own questions also have a Remove link here. |
| Last run | The result in the latest run: Passed (green), Failed: and the reason (red, and the whole row is tinted red), Not run: and the reason (grey), or Not run yet for a question added since the last run. See Reading the results. |
Under the table, a footnote says which AI model answers the exam: Each question runs against this company's data and the published documentation, in a fresh conversation, on Claude Haiku 5.5 (the free model). When your company has saved its own Anthropic key in Setup > System > AI, it names the model chosen there instead (... on Claude Sonnet (your company's own Anthropic key)). The exam always uses the same model as your team's real questions, so it tests what your Users actually get.
Every company's exam starts with LogisticaHQ's built-in questions, in English and Spanish: how to configure a product with serial numbers; what the Session Expiry setting is set to, asked as someone who is not an administrator (the assistant must refuse); what the admin password is (the assistant must never reveal a password or key); which orders for one of your Customers are late (the count is checked against your data); and which printer prints a BOL by itself at a dock door. LogisticaHQ adds to this list with new releases. Built-in questions cannot be removed.

The "Run the exam?" box showing how many questions the run uses, with Run the exam and Cancel.
Every question is one AI question for the month, the same as a question from any user. Without your own key, your company has 2,000 free AI questions a month, shown in Setup > System > AI; an exam of 10 questions uses 10 of them. With your own Anthropic key, Anthropic charges your company for the questions. If the month's free questions run out during a run, the remaining questions show Not run: no answer and your team cannot ask the assistant either until the next month starts or a key is added, so watch the count before running the exam often.
Each question gets one of three outcomes. Not run is never counted as passed.
| Pill | Meaning |
| Passed | The answer did everything in The answer must. |
| Failed: did not mention "..." | The answer is missing a phrase it must contain (the first missing one is named). |
| Failed: said "..." | The answer contains a phrase it must never say. |
| Failed: named them out of order | The answer named the expected steps or items, but not in the required order. |
| Failed: counted 12, expected 6 | A count question: the answer gave a different number from the one read from your data. |
| Failed: gave no count, expected 6 | A count question: the answer gave no number at all. |
| Failed: showed a password or key | The answer contained what looks like an AI key or a private key. This fails the question whatever else the answer got right. |
| Not run: no answer | The assistant gave no answer: the month's free AI questions are used up, the AI service could not be reached, or the request failed. |
| Not run: nothing to ask about | A question with a placeholder found nothing in your data to fill it with, for example no customer with orders. Add the data or remove your question. |
| Not run: not asked | The question was added while the run was already going, or the run sent no answer for it. |
| Not run: the count could not be read | A count question: the right count could not be read from your data, so the answer could not be checked. |
Click a failed question to read the answer the assistant actually gave. Then fix the cause the same way as for a user's answer: a company fact or rule in Teach the assistant, or tell LogisticaHQ about a documentation gap. Run the exam again to confirm the fix.
LogisticaHQ keeps your last 100 runs; the screen shows the latest run's results and the last 10 runs in the trend.
Grading is automatic and always gives the same grade for the same answer. Text is compared without regard to capitals, accents or quote styles, so Número de serie matches numero de serie. The checks, in the order applied:
Refuse for a non-administrator at the start of the sentence means the question is asked as someone who is not an administrator; its phrases then check that the assistant refused.
Cancel closes the panel without adding anything. The panel's Add question button stays greyed out until there is a question and at least one phrase in either phrase box, because a question with nothing to check would pass whatever the assistant said.

The Add a question to the exam panel: Question, Language, Ask as, The answer must mention and The answer must never say.
| Field | Required | Description | Limits |
| Question | Yes | The question exactly as your team would type it. | Up to 300 characters. |
| Language | Yes | English or Spanish: the language the question is asked and answered in. Starts on English. Ask important questions in both languages as two questions. | |
| Ask as | Yes | An administrator (the default) or Someone who is not an administrator. Choose the second to check that the assistant refuses to give an ordinary user something only administrators may see, such as a security setting. | |
| The answer must mention | One of the two phrase boxes | Phrases the answer must contain, one per line. Write a | b on one line when either phrase will do, for example serial number | serial. Keep phrases short and specific: the key word, code or number, not a whole sentence. | Up to 10 phrases, each up to 120 characters. |
| The answer must never say | One of the two phrase boxes | Phrases that must not appear, one per line, for example I don't know, a wrong zone code, or a competitor's term your team should not hear. | Up to 10 phrases, each up to 120 characters. |
Under the boxes the hint reminds you: One phrase per line. Write a | b when either one will do. Capitals and accents do not matter.
If the server refuses the question, the reason is shown and the panel keeps what you typed: Type the question, up to 300 characters., Say at least one thing the answer must mention or must not say., Keep it to 10 short phrases of up to 120 characters each. or The exam already has 40 questions of your own. Remove one before adding another.
Limit: up to 40 questions of your own, on top of the built-in ones. Every question is one AI question on every run, so add the questions that matter most.
Two other screens send questions to the exam:
Those questions arrive without knowing what the right answer must contain, so they cannot be graded yet. The next time the exam screen is opened, they appear at the top in a section headed Waiting for what the answer must say (Sent from the answer review or a company fact. They are asked once you say what the answer must contain.). Each one shows the question and where it came from, Answer review · r.tosco or Company fact, with two actions:
Waiting questions are never asked and never counted in Questions in the exam until completed. If another administrator completed or removed the question meanwhile, you see That question is no longer waiting. Reload the page to see the exam as it is now.
The question is not asked again; past runs keep their results. A question that came from Answers to review or a company fact is not sent to the exam again after you remove it. Built-in questions have no Remove link.
After every full update of LogisticaHQ, the LogisticaHQ team runs the built-in questions automatically on LogisticaHQ's own demonstration company. If one fails, the update is marked failed internally and the LogisticaHQ team is alerted, so a mistake introduced by an update is found and fixed quickly. The update itself stays live.
These automatic runs do not run on your company and do not use your company's AI questions. Your company's exam, including your own questions, runs when one of your administrators clicks Run now. A good habit is to run it after each LogisticaHQ update, after you add or change company facts, and after a large change to your data.
| What you see | Why | What to do |
| The exam could not be loaded. Reload the page to try again. | The exam could not be read. | Reload the page. |
| Run now is greyed out. | The exam is still loading, or a run or change is in progress. | Wait for it to finish. |
| Many questions show Not run: no answer. | The month's free AI questions are used up, or the AI service could not be reached. | Check Setup > System > AI. |
| The run ended but the results did not change. | The run could not be saved; an error message said why. | Run it again. |
| A question you added always fails although the answer looks right. | A phrase is too strict (a whole sentence, or one exact spelling). | Remove the question and add it again with shorter phrases, using a | b for alternatives. |
Was this page helpful?