espocrm-extensions/KnowledgeBase
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 14 Wiki Activity

feat(KB): v0.8.0 — bulk upload, AI classifier, labels, tool kind

Browse Source 
v0.8.0 ships the four-feature bundle from the v0.8.0 plan plus
mid-flight fixes discovered when batch-testing 5 circulars.

New capabilities:
  * Multi-file batch upload. AI classifier (Sonnet via ai-gateway)
    reads the first 3 pages of each PDF and proposes kind / title /
    identifier / labels / dates / summary. User reviews + edits per
    card before committing. Two-phase ingest: classify →
    awaiting_review → embed.
  * 'tool' kind for academic assessment instruments (GMFCS, MACS).
  * Labels (kb_label / kb_source_label) for sub-topic navigation;
    classifier proposes shared label slugs so the graph builds
    itself.
  * 'תוויות' admin sub-section for merge/delete of unused labels.
  * NEW 'ממתינים לאישור' panel: lists batches still in
    awaiting_review with status counters so a hard-refreshed user
    can resume their review — closes the RAM-only _activeBatch gap.

Mid-flight fixes folded in:
  * classifier timeout 45s→90s, concurrency 5→2. ai-gateway
    serializes through a single Claude OAuth session; with conc=5,
    jobs 4-5 of a 5-file batch consistently timed out.
  * labels schema array-of-string → comma-separated string. Claude
    via ai-gateway returned [{}, {}, {}] for items:{type:"string"}.
    _normalize accepts both shapes defensively.

Backfilled ai_classified_at + description + 1-3 labels for the 10
sources ingested in earlier sessions so the filter UX is uniform.

Backend (shira-hermes) in commits 0d678da (Phase 6) + bb23b8a
(this session's fixes).

Refs Task Master #20

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chaim Marcus
2026-04-25 20:18:34 +00:00
parent 30a9c397e8
commit 8a4f16a50f
7 changed files with 1020 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
files/client/custom/modules/knowledge-base/res/templates/kb/index.tpl
+57 -51
View File
@@ -37,6 +37,7 @@
<option value="regulation">תקנות</option>
<option value="circular">חוזרים</option>
<option value="caselaw">פסיקה</option>
<option value="tool">כלי הערכה</option>
</select>
<button type="button" class="btn btn-primary" data-action="submit">חפש</button>
<button type="button" class="btn btn-default" data-action="clearResults"
@@ -99,6 +100,7 @@
<option value="regulation">תקנות</option>
<option value="circular">חוזרים</option>
<option value="caselaw">פסיקה</option>
<option value="tool">כלי הערכה</option>
</select>
<button type="button" class="btn btn-default btn-sm" data-action="refreshSources" title="רענן">
<span class="glyphicon glyphicon-refresh"></span>
@@ -109,62 +111,66 @@
<div class="text-muted">טוען מקורות…</div>
</div>
</div>
<div class="panel panel-default" style="padding:12px;">
<h4 style="margin-top:0;">העלאת מסמך חדש</h4>
<form class="kb-upload-form" enctype="multipart/form-data"
style="display:flex;flex-direction:column;gap:10px;">
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label for="kb-upload-file" class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">קובץ:</label>
<input type="file" id="kb-upload-file" name="file"
accept=".pdf,.docx,.txt"
style="flex:1 1 280px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label for="kb-upload-kind" class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">סוג:</label>
<select id="kb-upload-kind" class="form-control" name="kind"
style="flex:0 0 auto;width:200px;">
<div class="panel panel-default kb-labels-panel" style="padding:12px;">
<div style="display:flex;justify-content:space-between;align-items:center;flex-wrap:wrap;gap:8px;margin-bottom:8px;">
<h4 style="margin:0;">תוויות תת-נושא</h4>
<button type="button" class="btn btn-default btn-sm" data-action="refreshLabels" title="רענן">
<span class="glyphicon glyphicon-refresh"></span>
</button>
</div>
<div class="kb-labels-table">
<div class="text-muted">טוען תוויות…</div>
</div>
<div class="text-muted small" style="margin-top:6px;">
תוויות נוצרות אוטומטית כשמעלים מסמך חדש ושירה מציעה תת-נושא. כאן ניתן למחוק תוויות שאינן בשימוש.
</div>
</div>
<div class="panel panel-default kb-pending-panel" style="padding:12px;">
<div style="display:flex;justify-content:space-between;align-items:center;flex-wrap:wrap;gap:8px;margin-bottom:8px;">
<h4 style="margin:0;">ממתינים לאישור<span class="kb-pending-count text-muted" style="font-weight:normal;margin-right:6px;"></span></h4>
<button type="button" class="btn btn-default btn-sm" data-action="refreshPending" title="רענן">
<span class="glyphicon glyphicon-refresh"></span>
</button>
</div>
<div class="kb-pending-list">
<div class="text-muted">טוען…</div>
</div>
<div class="text-muted small" style="margin-top:6px;">
באצ׳ים שעלו וממתינים לעריכה ואישור. לחץ "פתח" כדי לחזור למסך הסקירה.
</div>
</div>
<div class="panel panel-default kb-batch-panel" style="padding:12px;">
<div style="display:flex;justify-content:space-between;align-items:center;flex-wrap:wrap;gap:8px;margin-bottom:8px;">
<h4 style="margin:0;">העלאת מסמכים</h4>
<div style="display:flex;gap:8px;align-items:center;">
<select id="kb-batch-kind" class="form-control" style="width:auto;">
<option value="circular" selected>חוזר</option>
<option value="law">חוק</option>
<option value="regulation">תקנות</option>
<option value="circular">חוזר</option>
<option value="caselaw">פסיקה</option>
<option value="tool">כלי הערכה</option>
</select>
<input type="file" id="kb-batch-files" multiple
accept=".pdf,.docx,.txt" style="display:none;" />
<button type="button" class="btn btn-primary btn-sm" data-action="pickBatchFiles">
<span class="glyphicon glyphicon-cloud-upload"></span> בחר קבצים…
</button>
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label for="kb-upload-title" class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">כותרת:</label>
<input type="text" id="kb-upload-title" class="form-control" name="title"
placeholder="ברירת מחדל: שם הקובץ"
style="flex:1 1 280px;" />
</div>
<details>
<summary class="text-muted" style="cursor:pointer;">מטא-דאטה נוסף (אופציונלי)</summary>
<div style="display:flex;flex-direction:column;gap:8px;margin-top:8px;padding-right:12px;">
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">מזהה:</label>
<input type="text" class="form-control" name="identifier"
placeholder="למשל: ח'(353) 14.1.2018"
style="flex:1 1 280px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">פורסם ב:</label>
<input type="date" class="form-control" name="published_at"
style="flex:0 0 auto;width:200px;" />
<label class="text-muted" style="margin:0;font-weight:normal;">תוקף מ:</label>
<input type="date" class="form-control" name="effective_at"
style="flex:0 0 auto;width:200px;" />
</div>
<div style="display:flex;gap:8px;align-items:center;flex-wrap:wrap;">
<label class="text-muted" style="margin:0;font-weight:normal;min-width:80px;">קישור מקור:</label>
<input type="url" class="form-control" name="source_url"
placeholder="https://…"
style="flex:1 1 280px;" />
</div>
</div>
</details>
<div style="display:flex;gap:8px;align-items:center;">
<button type="button" class="btn btn-primary" data-action="submitUpload">העלה</button>
<span class="kb-upload-hint text-muted small">מקסימום 50MB. PDF / DOCX / TXT.</span>
</div>
</form>
</div>
<div class="kb-batch-hint text-muted small" style="margin-bottom:8px;">
ניתן לבחור מספר קבצים בו-זמנית. שירה תנתח כל קובץ ותציע מטא-דאטה (סוג / כותרת / תוויות / סיכום) — תוכל לערוך לפני אישור הטמעה.
מקסימום 50MB לקובץ, עד 50 קבצים בקבוצה.
</div>
<div class="kb-batch-cards" style="display:flex;flex-direction:column;gap:10px;"></div>
<div class="kb-batch-actions" style="display:none;justify-content:space-between;align-items:center;margin-top:12px;padding-top:12px;border-top:1px solid #eee;">
<button type="button" class="btn btn-default btn-sm" data-action="discardAllBatch">בטל הכל</button>
<button type="button" class="btn btn-primary" data-action="commitAllBatch">
אישור והטמעה
<span class="kb-batch-commit-count"></span>
</button>
</div>
</div>
<div class="panel panel-default" style="padding:12px;">
files/client/custom/modules/knowledge-base/src/views/kb/index.js
+563 -6
View File
@@ -17,6 +17,9 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
let _pollIntervalId = null; // setInterval handle so remounts don't double-poll
let _adminSourcesByTopic = {}; // {topicId: [...sources]} cache for the manage tab
let _adminTopics = null; // [...topics admin view] cache (incl. is_active=false)
let _adminLabels = null; // labels admin cache
let _activeBatch = null; // {batchId, kind, cards:[{jobId, filename, status, ai, edits, labels}]}
let _batchPollIntervalId = null;
const SS_ASK = 'kb-last-ask';
const SS_SEARCH = 'kb-last-search';
@@ -190,6 +193,49 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
const id = parseInt($(e.currentTarget).data('id'), 10);
if (id) this._deleteTopic(id);
},
// ── Phase 6 (v0.8.0): batch upload + labels admin ──────────────
'click [data-action="pickBatchFiles"]': function (e) {
e.preventDefault();
this.$el.find('#kb-batch-files').trigger('click');
},
'change #kb-batch-files': function (e) {
const files = Array.from(e.target.files || []);
if (files.length) this._uploadBatch(files);
e.target.value = '';
},
'click [data-action="discardCard"]': function (e) {
e.preventDefault();
const jobId = parseInt($(e.currentTarget).data('jobid'), 10);
if (jobId) this._discardCard(jobId);
},
'click [data-action="commitAllBatch"]': function (e) {
e.preventDefault();
this._commitAllBatch();
},
'click [data-action="discardAllBatch"]': function (e) {
e.preventDefault();
this._discardAllBatch();
},
'click [data-action="refreshLabels"]': function (e) {
e.preventDefault();
this._loadAdminLabels(/*force*/ true);
},
'click [data-action="deleteLabel"]': function (e) {
e.preventDefault();
const id = parseInt($(e.currentTarget).data('id'), 10);
if (id) this._deleteLabel(id);
},
'click [data-action="refreshPending"]': function (e) {
e.preventDefault();
this._loadPendingBatches();
},
'click [data-action="openBatch"]': function (e) {
e.preventDefault();
const batchId = $(e.currentTarget).data('batchid');
const topicAttr = $(e.currentTarget).data('topicid');
const topicId = (topicAttr === '' || topicAttr == null) ? null : parseInt(topicAttr, 10);
if (batchId) this._openBatch(String(batchId), topicId);
},
},
// Cancel any in-flight ask/search, drop the cached last-result for
@@ -285,6 +331,17 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
this._loadAdminSources();
// Phase 4: load the topics admin table (firm-wide, not topic-scoped).
this._loadAdminTopics();
// Phase 6: load the labels admin table.
this._loadAdminLabels();
// Phase 6: load batches that still need review (RAM-only
// _activeBatch is wiped by hard refresh; this lets the user
// resume any pending batch from the panel).
this._loadPendingBatches();
// Phase 6: redraw any in-flight batch upload (survives remount).
if (_activeBatch) {
this._renderBatchCards();
if (this._batchHasClassifying()) this._startBatchPolling();
}
},
_loadJobsList: function () {
@@ -317,7 +374,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
done: '<span class="label label-success">הושלם</span>',
failed: '<span class="label label-danger">נכשל</span>',
};
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזר', caselaw: 'פסיקה'};
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזר', caselaw: 'פסיקה', tool: 'כלי הערכה'};
const rows = items.map(j => {
const when = j.completed_at || j.started_at || j.created_at || '';
const whenShort = String(when).slice(0, 19).replace('T', ' ');
@@ -618,7 +675,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
return;
}
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזר', caselaw: 'פסיקה'};
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזר', caselaw: 'פסיקה', tool: 'כלי הערכה'};
const rows = items.map(s => {
const li = s.last_ingest || {};
const failedFlag = li.status === 'failed'
@@ -772,6 +829,506 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
});
},
// ── Phase 6 (v0.8.0): labels admin ─────────────────────────────────
_loadAdminLabels: function (force) {
const $table = this.$el.find('.kb-labels-table');
if (!$table.length) return;
if (!force && _adminLabels) {
this._renderLabelsTable();
return;
}
const self = this;
Espo.Ajax.getRequest('KnowledgeBase/action/labels', {
topicId: this.topicId,
limit: 200,
}).then(res => {
_adminLabels = res && res.items || [];
self._renderLabelsTable();
}).catch(err => {
console.error('KB: load labels failed', err);
$table.html('<div class="text-muted">שגיאה בטעינת תוויות.</div>');
});
},
_renderLabelsTable: function () {
const $table = this.$el.find('.kb-labels-table');
if (!$table.length) return;
const items = _adminLabels || [];
if (!items.length) {
$table.html('<div class="text-muted">אין תוויות עדיין. הן ייווצרו אוטומטית מההצעות של שירה בעת ההעלאה.</div>');
return;
}
const rows = items.map(l => `<tr data-id="${l.id}">
<td><span style="font-weight:500;">${this.escape(l.name)}</span></td>
<td><code class="text-muted small" style="direction:ltr;">${this.escape(l.slug)}</code></td>
<td class="text-center">${l.usage_count || 0}</td>
<td style="text-align:left;">
${(l.usage_count || 0) === 0
? `<button class="btn btn-link btn-sm" data-action="deleteLabel" data-id="${l.id}" title="מחק" style="color:#d9534f;"><span class="glyphicon glyphicon-trash"></span></button>`
: '<span class="text-muted small">בשימוש</span>'}
</td>
</tr>`).join('');
$table.html(`
<table class="table table-condensed table-hover" style="margin:0;">
<thead>
<tr><th>שם תווית</th><th style="width:200px;">slug</th><th style="width:80px;text-align:center;">שימושים</th><th style="width:80px;"></th></tr>
</thead>
<tbody>${rows}</tbody>
</table>
`);
},
_deleteLabel: function (id) {
const lbl = (_adminLabels || []).find(l => l.id === id);
if (!lbl) return;
if (!confirm(`למחוק את התווית "${lbl.name}" (${lbl.slug})?`)) return;
const self = this;
Espo.Ajax.postRequest('KnowledgeBase/action/deleteLabel', {id})
.then(() => {
_adminLabels = (_adminLabels || []).filter(l => l.id !== id);
self._renderLabelsTable();
})
.catch(err => {
let msg = 'שגיאה לא ידועה';
try { const j = JSON.parse(err && err.responseText || '{}'); msg = j.message || j.detail || msg; } catch (e) {}
alert('מחיקה נכשלה: ' + String(msg).slice(0, 200));
});
},
// ── Phase 6: batch upload + AI review ──────────────────────────────
// Pending-review panel: resumes any batch the user uploaded
// earlier and didn't commit. _activeBatch is RAM-only, so a hard
// refresh wipes the review screen — without this list the only
// recovery path was the DB.
_loadPendingBatches: function () {
const self = this;
const $list = this.$el.find('.kb-pending-list');
const $count = this.$el.find('.kb-pending-count');
if (!$list.length) return;
Espo.Ajax.getRequest('KnowledgeBase/action/pendingBatches', {limit: 50})
.then(res => {
const items = (res && res.items) || [];
self._renderPendingBatches(items);
$count.text(items.length ? ` (${items.length})` : '');
})
.catch(err => {
console.error('KB: failed to load pending batches', err);
$list.html('<div class="text-muted">שגיאה בטעינת ממתינים לאישור.</div>');
});
},
_renderPendingBatches: function (items) {
const $list = this.$el.find('.kb-pending-list');
if (!$list.length) return;
if (!items.length) {
$list.html('<div class="text-muted">אין באצ׳ים ממתינים לאישור.</div>');
return;
}
const kindHe = {
law: 'חוק', regulation: 'תקנות', circular: 'חוזרים',
caselaw: 'פסיקה', tool: 'כלי הערכה',
};
const escape = this.escape.bind(this);
const html = items.map(b => {
const kinds = (b.kinds || []).map(k => kindHe[k] || k).join(', ');
const time = b.created_at ? new Date(b.created_at).toLocaleString('he-IL') : '';
const counters = [];
if (b.awaiting_review) counters.push(`<span class="label label-success">${b.awaiting_review} מוכנים</span>`);
if (b.processing) counters.push(`<span class="label label-info">${b.processing} בתהליך</span>`);
if (b.queued) counters.push(`<span class="label label-default">${b.queued} בתור</span>`);
if (b.failed) counters.push(`<span class="label label-danger">${b.failed} נכשלו</span>`);
if (b.done) counters.push(`<span class="label label-success">${b.done} הושלמו</span>`);
const isCurrent = _activeBatch && _activeBatch.batchId === b.batch_id;
const more = b.total > 1 ? ` <span class="text-muted">+ ${b.total - 1} נוספים</span>` : '';
return `<div class="kb-pending-row" style="display:flex;justify-content:space-between;align-items:center;gap:10px;padding:8px;border-bottom:1px solid #eee;">
<div style="flex:1;min-width:0;">
<div style="font-weight:500;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;">📦 ${escape(b.first_filename || '(ללא שם)')}${more}</div>
<div class="text-muted small">${escape(kinds)} · ${escape(time)} · ${counters.join(' ')}</div>
</div>
<button type="button" class="btn btn-${isCurrent ? 'default' : 'primary'} btn-sm" data-action="openBatch" data-batchid="${escape(b.batch_id)}" data-topicid="${b.topic_id || ''}"${isCurrent ? ' disabled title="פתוח כרגע"' : ''}>${isCurrent ? 'פתוח' : 'פתח'}</button>
</div>`;
}).join('');
$list.html(html);
},
_openBatch: function (batchId, topicId) {
const self = this;
if (!batchId) return;
if (_activeBatch && _activeBatch.batchId === batchId) {
this._scrollToBatch();
return;
}
// If the batch belongs to a different topic, switch — labels
// and source views are topic-scoped, the user should see the
// matching context.
if (topicId && this.topicId !== topicId) {
this.switchTopic(topicId);
}
this._stopBatchPolling();
Espo.Ajax.getRequest('KnowledgeBase/action/batch', {batchId})
.then(res => {
const items = (res && res.items) || [];
if (!items.length) {
alert('הבאצ׳ ריק או נמחק.');
return;
}
const cards = items.map(it => {
let ai = it.ai_suggestions;
if (typeof ai === 'string') {
try { ai = JSON.parse(ai); } catch (e) { ai = {}; }
}
let s = it.status;
if (s === 'processing' && it.processing_stage === 'classifying') s = 'classifying';
else if (s === 'processing' && it.processing_stage === 'embedding') s = 'committing';
return {
jobId: it.id,
filename: it.original_filename,
status: s,
ai: (ai && Object.keys(ai).length) ? ai : null,
edits: null,
error: it.error_message || null,
};
});
_activeBatch = {
batchId: batchId,
kind: (items[0] && items[0].kind) || 'circular',
topicId: topicId || self.topicId,
cards,
};
self._renderBatchCards();
// Repaint pending list so the opened row shows "פתוח" instead of "פתח".
self._loadPendingBatches();
self._scrollToBatch();
if (self._batchHasClassifying() || self._batchHasCommitting()) {
self._startBatchPolling();
}
})
.catch(err => {
console.error('KB: failed to open batch', err);
alert('פתיחת הבאצ׳ נכשלה.');
});
},
_scrollToBatch: function () {
const $batch = this.$el.find('.kb-batch-panel');
if ($batch.length && $batch[0].scrollIntoView) {
$batch[0].scrollIntoView({behavior: 'smooth', block: 'start'});
}
},
_uploadBatch: function (fileList) {
const $kindSel = this.$el.find('#kb-batch-kind');
const kind = $kindSel.val() || 'circular';
const topicId = this.topicId;
if (topicId == null) {
alert('בחר נושא לפני העלאה.');
return;
}
// Filter on size + extension client-side; server enforces too.
const ok = [];
for (const f of fileList) {
if (!/\.(pdf|docx|txt)$/i.test(f.name)) continue;
if (f.size > 50 * 1024 * 1024) {
alert(`הקובץ "${f.name}" גדול מ-50MB ולא יועלה.`);
continue;
}
ok.push(f);
}
if (!ok.length) return;
const fd = new FormData();
fd.append('kind', kind);
fd.append('topicId', String(topicId));
// PHP-side $_FILES['files'] requires the bracket suffix to
// recognize repeated fields as a multi-file array. The
// EspoCRM controller un-array's into a plain `files` field
// before forwarding to shira-hermes (FastAPI standard).
for (const f of ok) fd.append('files[]', f, f.name);
// Initialize cards in placeholder state. They get the real
// job_id from the server response.
_activeBatch = {
batchId: null,
kind,
topicId,
cards: ok.map(f => ({
jobId: null,
filename: f.name,
status: 'uploading',
ai: null,
edits: null,
})),
};
this._renderBatchCards();
const self = this;
$.ajax({
url: 'api/v1/KnowledgeBase/action/uploadBatch',
method: 'POST',
data: fd,
contentType: false,
processData: false,
cache: false,
timeout: 300000,
}).then(res => {
if (!_activeBatch) return;
_activeBatch.batchId = res.batch_id;
// Match returned jobs to local cards by filename order.
(res.jobs || []).forEach((j, i) => {
if (_activeBatch.cards[i]) {
_activeBatch.cards[i].jobId = j.job_id || null;
_activeBatch.cards[i].status = j.error ? 'failed' : 'queued';
if (j.error) _activeBatch.cards[i].error = j.error;
}
});
self._renderBatchCards();
self._startBatchPolling();
}).fail(xhr => {
let msg = 'שגיאה לא ידועה';
try {
const j = JSON.parse(xhr.responseText || '{}');
msg = j.message || j.detail || xhr.responseText || msg;
} catch (e) { msg = xhr.responseText || msg; }
alert('העלאה נכשלה: ' + String(msg).slice(0, 300));
_activeBatch = null;
self._renderBatchCards();
});
},
_renderBatchCards: function () {
const $cards = this.$el.find('.kb-batch-cards');
const $actions = this.$el.find('.kb-batch-actions');
if (!$cards.length) return;
if (!_activeBatch || !_activeBatch.cards.length) {
$cards.empty();
$actions.hide();
return;
}
const statusLabel = (s) => ({
uploading: '<span class="label label-default">מעלה…</span>',
queued: '<span class="label label-default">בתור</span>',
classifying: '<span class="label label-info">מנתח מטא-דאטה…</span>',
awaiting_review: '<span class="label label-success">[✓ מוכן]</span>',
committing: '<span class="label label-info">מטמיע…</span>',
done: '<span class="label label-success">[✓ הושלם]</span>',
failed: '<span class="label label-danger">[⚠️ נכשל]</span>',
}[s] || `<span class="label label-default">${this.escape(s)}</span>`);
const kindOptions = (selected) => {
const opts = [
['law', 'חוק'],
['regulation', 'תקנות'],
['circular', 'חוזר'],
['caselaw', 'פסיקה'],
['tool', 'כלי הערכה'],
];
return opts.map(([v, l]) =>
`<option value="${v}"${v === selected ? ' selected' : ''}>${l}</option>`
).join('');
};
let readyCount = 0;
const html = _activeBatch.cards.map((c, idx) => {
if (c.status === 'awaiting_review') readyCount++;
const ai = c.ai || {};
const e = c.edits || {};
const val = (k) => this.escape(e[k] != null ? e[k] : (ai[k] || ''));
const labelsStr = e.labelsStr != null ? e.labelsStr : ((ai.labels || []).join(', '));
let body = '';
if (c.status === 'failed') {
body = `<div class="text-danger small">${this.escape(c.error || 'שגיאה')}</div>`;
} else if (c.status === 'awaiting_review' || c.status === 'committing' || c.status === 'done') {
const disabled = c.status !== 'awaiting_review' ? ' disabled' : '';
body = `
<div style="display:flex;gap:8px;flex-wrap:wrap;">
<select class="form-control" data-card="${idx}" data-edit="kind" style="width:auto;"${disabled}>${kindOptions(e.kind || ai.kind || _activeBatch.kind)}</select>
<input type="text" class="form-control" data-card="${idx}" data-edit="title" value="${val('title')}" placeholder="כותרת" style="flex:1 1 240px;"${disabled} />
</div>
<div style="display:flex;gap:8px;flex-wrap:wrap;">
<input type="text" class="form-control" data-card="${idx}" data-edit="identifier" value="${val('identifier')}" placeholder="מזהה (אופציונלי)" style="flex:1 1 200px;"${disabled} />
<input type="date" class="form-control" data-card="${idx}" data-edit="published_at" value="${this.escape((e.published_at != null ? e.published_at : (ai.published_at || '')).slice(0,10))}" style="width:160px;"${disabled} />
</div>
<input type="text" class="form-control" data-card="${idx}" data-edit="labelsStr" value="${this.escape(labelsStr)}" placeholder="תוויות (מופרדות בפסיק)" style="width:100%;"${disabled} />
<textarea class="form-control" rows="2" data-card="${idx}" data-edit="summary" placeholder="סיכום קצר" style="width:100%;"${disabled}>${this.escape(e.summary != null ? e.summary : (ai.summary || ''))}</textarea>
`;
} else {
body = '<div class="text-muted small">ממתין לניתוח…</div>';
}
return `<div class="kb-batch-card panel" data-card="${idx}" style="padding:10px;">
<div style="display:flex;justify-content:space-between;align-items:center;gap:8px;margin-bottom:6px;">
<div style="font-weight:500;">📄 ${this.escape(c.filename)}</div>
<div style="display:flex;gap:8px;align-items:center;">
${statusLabel(c.status)}
${c.status !== 'done' && c.status !== 'committing' ? `<button class="btn btn-link btn-sm" data-action="discardCard" data-jobid="${c.jobId || 0}" title="הסר" style="color:#d9534f;"><span class="glyphicon glyphicon-remove"></span></button>` : ''}
</div>
</div>
<div style="display:flex;flex-direction:column;gap:6px;">${body}</div>
</div>`;
}).join('');
$cards.html(html);
$actions.toggle(_activeBatch.cards.length > 0);
this.$el.find('.kb-batch-commit-count').text(readyCount > 0 ? `(${readyCount})` : '');
this.$el.find('[data-action="commitAllBatch"]').prop('disabled', readyCount === 0);
// Wire input change → save into card.edits
const self = this;
$cards.find('[data-edit]').off('input.batch change.batch').on('input.batch change.batch', function () {
const idx = parseInt($(this).data('card'), 10);
const k = $(this).data('edit');
if (_activeBatch && _activeBatch.cards[idx]) {
if (!_activeBatch.cards[idx].edits) _activeBatch.cards[idx].edits = {};
_activeBatch.cards[idx].edits[k] = $(this).val();
}
});
},
_batchHasClassifying: function () {
return _activeBatch && _activeBatch.cards.some(c =>
c.status === 'queued' || c.status === 'classifying' || c.status === 'uploading'
);
},
_startBatchPolling: function () {
this._stopBatchPolling();
const self = this;
// Refresh the pending list once at start so a freshly-uploaded
// batch appears there right away (without waiting for the
// batch to quiesce).
this._loadPendingBatches();
const tick = () => {
if (!_activeBatch || !_activeBatch.batchId) return;
Espo.Ajax.getRequest('KnowledgeBase/action/batch', {batchId: _activeBatch.batchId})
.then(res => {
if (!_activeBatch) return;
const byId = {};
for (const it of res.items || []) byId[it.id] = it;
for (const card of _activeBatch.cards) {
if (!card.jobId) continue;
const it = byId[card.jobId];
if (!it) continue;
// Map server status → card status
let s = it.status;
if (s === 'processing' && it.processing_stage === 'classifying') s = 'classifying';
else if (s === 'processing' && it.processing_stage === 'embedding') s = 'committing';
card.status = s;
card.error = it.error_message || null;
// Parse ai_suggestions if string
let ai = it.ai_suggestions;
if (typeof ai === 'string') {
try { ai = JSON.parse(ai); } catch (e) { ai = {}; }
}
if (ai && Object.keys(ai).length) card.ai = ai;
}
self._renderBatchCards();
if (!self._batchHasClassifying() && !self._batchHasCommitting()) {
self._stopBatchPolling();
// Refresh sources/jobs/labels list since new ones may have landed.
if (_activeBatch.cards.some(c => c.status === 'done')) {
self._loadJobsList();
self._loadAdminSources(true);
self._loadAdminLabels(true);
}
// Pending list reflects awaiting_review counters
// — refresh on every batch quiescence so commits
// and discards drop the row out cleanly.
self._loadPendingBatches();
}
})
.catch(err => console.error('KB: batch poll failed', err));
};
tick();
_batchPollIntervalId = setInterval(tick, 3000);
},
_stopBatchPolling: function () {
if (_batchPollIntervalId) {
clearInterval(_batchPollIntervalId);
_batchPollIntervalId = null;
}
},
_batchHasCommitting: function () {
return _activeBatch && _activeBatch.cards.some(c => c.status === 'committing');
},
_commitAllBatch: function () {
if (!_activeBatch) return;
const self = this;
const ready = _activeBatch.cards.filter(c => c.status === 'awaiting_review' && c.jobId);
if (!ready.length) return;
for (const card of ready) {
const ai = card.ai || {};
const e = card.edits || {};
const labelsStr = e.labelsStr != null ? e.labelsStr : ((ai.labels || []).join(', '));
const newLabels = labelsStr
.split(',')
.map(s => s.trim())
.filter(s => s.length);
const payload = {
id: card.jobId,
kind: e.kind || ai.kind || _activeBatch.kind,
title: (e.title != null ? e.title : (ai.title || card.filename)).trim() || card.filename,
identifier: (e.identifier != null ? e.identifier : (ai.identifier || '')).trim() || null,
published_at: (e.published_at != null ? e.published_at : (ai.published_at || '')) || null,
effective_at: (e.effective_at != null ? e.effective_at : (ai.effective_at || '')) || null,
source_url: (e.source_url != null ? e.source_url : (ai.source_url || '')) || null,
summary: (e.summary != null ? e.summary : (ai.summary || '')) || null,
label_slugs: [],
new_labels: newLabels,
};
card.status = 'committing';
Espo.Ajax.postRequest('KnowledgeBase/action/commitJob', payload)
.catch(err => {
console.error('KB: commit failed for job', card.jobId, err);
card.status = 'failed';
card.error = (err && err.responseText) || 'commit failed';
self._renderBatchCards();
});
}
self._renderBatchCards();
self._startBatchPolling();
},
_discardCard: function (jobId) {
if (!_activeBatch) return;
const idx = _activeBatch.cards.findIndex(c => c.jobId === jobId);
if (idx < 0) return;
// If never queued (server failed before assigning jobId), just drop locally.
if (!jobId) {
_activeBatch.cards.splice(idx, 1);
if (!_activeBatch.cards.length) _activeBatch = null;
this._renderBatchCards();
return;
}
const self = this;
Espo.Ajax.postRequest('KnowledgeBase/action/discardJob', {id: jobId})
.finally(() => {
_activeBatch.cards.splice(idx, 1);
if (!_activeBatch.cards.length) _activeBatch = null;
self._renderBatchCards();
});
},
_discardAllBatch: function () {
if (!_activeBatch) return;
if (!confirm(`לבטל את כל ${_activeBatch.cards.length} הקבצים בקבוצה?`)) return;
const self = this;
const promises = _activeBatch.cards
.filter(c => c.jobId && c.status !== 'done')
.map(c => Espo.Ajax.postRequest('KnowledgeBase/action/discardJob', {id: c.jobId})
.catch(() => null));
Promise.all(promises).finally(() => {
_activeBatch = null;
self._stopBatchPolling();
self._renderBatchCards();
self._loadJobsList();
});
},
// Live status panel rendered above the upload form during processing.
_renderJobStatus: function (job) {
const $form = this.$el.find('.kb-upload-form');
@@ -1319,7 +1876,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
return;
}
const initialIdx = (typeof selectedIdx === 'number' && hits[selectedIdx]) ? selectedIdx : 0;
const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר', caselaw: 'פסיקה'};
const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר', caselaw: 'פסיקה', tool: 'כלי'};
const self = this;
const isPdfBacked = h => h && (h.kind === 'regulation' || h.kind === 'circular');
@@ -1618,7 +2175,7 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
answerHtml = '<div style="white-space:pre-wrap;">' + this.escape(text || '') + '</div>';
}
const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר', caselaw: 'פסיקה'};
const kindHe = {law: 'חוק', regulation: 'תקנה', circular: 'חוזר', caselaw: 'פסיקה', tool: 'כלי'};
// Dedup sources by source_id (first occurrence = most relevant),
// but remember every page that came up per source so users can
@@ -1758,8 +2315,8 @@ define('modules/knowledge-base/views/kb/index', ['view'], function (Dep) {
$list.html('<div class="text-muted">אין מקורות.</div>');
return;
}
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזרים', caselaw: 'פסיקה'};
const grouped = {law: [], regulation: [], circular: [], caselaw: []};
const kindHe = {law: 'חוק', regulation: 'תקנות', circular: 'חוזרים', caselaw: 'פסיקה', tool: 'כלי הערכה'};
const grouped = {law: [], regulation: [], circular: [], caselaw: [], tool: []};
this._sources.forEach(s => {
if (grouped[s.kind]) grouped[s.kind].push(s);
});
files/custom/Espo/Modules/KnowledgeBase/Controllers/KnowledgeBase.php
+157
View File
@@ -307,4 +307,161 @@ class KnowledgeBase
}
return $this->getService()->deleteAdminTopic((int) $id);
}
// ── Phase 6 (v0.8.0): bulk upload + AI classifier + labels ──────────────
public function postActionUploadBatch(Request $request, Response $response): array
{
$this->checkAccess();
// Multi-file: $_FILES['files'] is an array-of-arrays in PHP when
// the form submits files[] repeated. Normalize per-file.
$rawFiles = $_FILES['files'] ?? null;
if (!$rawFiles || !is_array($rawFiles['tmp_name'] ?? null)) {
throw new BadRequest('No files uploaded (form field must be `files[]`).');
}
$files = [];
$count = count($rawFiles['tmp_name']);
for ($i = 0; $i < $count; $i++) {
$err = $rawFiles['error'][$i] ?? UPLOAD_ERR_NO_FILE;
$tmp = $rawFiles['tmp_name'][$i] ?? '';
if ($err !== UPLOAD_ERR_OK || !$tmp || !is_uploaded_file($tmp)) {
continue; // server-side filter; client filters too
}
$files[] = [
'tmp_name' => $tmp,
'name' => (string) ($rawFiles['name'][$i] ?? 'upload'),
'type' => (string) ($rawFiles['type'][$i] ?? 'application/octet-stream'),
'size' => (int) ($rawFiles['size'][$i] ?? 0),
];
}
if (!$files) {
throw new BadRequest('No valid files in upload.');
}
$kind = $_POST['kind'] ?? null;
if (!in_array($kind, ['law', 'regulation', 'circular', 'caselaw', 'tool'], true)) {
throw new BadRequest('kind must be one of: law, regulation, circular, caselaw, tool.');
}
$topicId = $this->coerceTopicId($_POST['topicId'] ?? $_POST['topic_id'] ?? null);
if ($topicId === null) {
throw new BadRequest('topicId is required.');
}
return $this->getService()->uploadBatch(
$files,
(string) $kind,
$topicId,
(string) $this->user->get('userName'),
);
}
public function getActionBatch(Request $request, Response $response): array
{
$this->checkAccess();
$batchId = $request->getQueryParam('batchId');
if (!$batchId) {
throw new BadRequest('batchId is required.');
}
return $this->getService()->getBatch((string) $batchId);
}
public function getActionPendingBatches(Request $request, Response $response): array
{
$this->checkAccess();
$limit = (int) ($request->getQueryParam('limit') ?? 50);
if ($limit < 1) $limit = 1;
if ($limit > 200) $limit = 200;
return $this->getService()->listPendingBatches($limit);
}
public function postActionCommitJob(Request $request, Response $response): array
{
$this->checkAccess();
$body = $request->getParsedBody();
$id = $body->id ?? null;
if (!$id || !is_numeric($id)) {
throw new BadRequest('id (numeric) is required.');
}
$kind = $body->kind ?? null;
if (!in_array($kind, ['law', 'regulation', 'circular', 'caselaw', 'tool'], true)) {
throw new BadRequest('kind must be one of: law, regulation, circular, caselaw, tool.');
}
$payload = [
'kind' => $kind,
'title' => isset($body->title) ? trim((string) $body->title) : '',
'identifier' => isset($body->identifier) ? trim((string) $body->identifier) : null,
'source_url' => $body->source_url ?? $body->sourceUrl ?? null,
'published_at' => $body->published_at ?? $body->publishedAt ?? null,
'effective_at' => $body->effective_at ?? $body->effectiveAt ?? null,
'summary' => isset($body->summary) ? trim((string) $body->summary) : null,
'label_slugs' => is_array($body->label_slugs ?? null) ? $body->label_slugs : [],
'new_labels' => is_array($body->new_labels ?? null) ? $body->new_labels : [],
];
return $this->getService()->commitJob(
(int) $id, $payload, (string) $this->user->get('userName')
);
}
public function postActionDiscardJob(Request $request, Response $response): array
{
$this->checkAccess();
$body = $request->getParsedBody();
$id = $body->id ?? null;
if (!$id || !is_numeric($id)) {
throw new BadRequest('id (numeric) is required.');
}
return $this->getService()->discardJob(
(int) $id, (string) $this->user->get('userName')
);
}
public function getActionLabels(Request $request, Response $response): array
{
$this->checkAccess();
$topicId = $this->coerceTopicId($request->getQueryParam('topicId'));
$q = $request->getQueryParam('q');
$limit = $request->getQueryParam('limit');
$limitInt = ($limit !== null && is_numeric($limit)) ? (int) $limit : 50;
return $this->getService()->listLabels($topicId, $q, $limitInt);
}
public function postActionCreateLabel(Request $request, Response $response): array
{
$this->checkAccess();
$body = $request->getParsedBody();
if (empty($body->slug) || empty($body->name)) {
throw new BadRequest('slug and name are required.');
}
return $this->getService()->createLabel([
'slug' => trim((string) $body->slug),
'name' => trim((string) $body->name),
'topic_id' => isset($body->topic_id) ? (int) $body->topic_id : null,
], (string) $this->user->get('userName'));
}
public function postActionMergeLabels(Request $request, Response $response): array
{
$this->checkAccess();
$body = $request->getParsedBody();
$id = $body->id ?? null;
$intoId = $body->into_label_id ?? $body->intoLabelId ?? null;
if (!$id || !is_numeric($id) || !$intoId || !is_numeric($intoId)) {
throw new BadRequest('id and into_label_id (numeric) are required.');
}
return $this->getService()->mergeLabels((int) $id, (int) $intoId);
}
public function postActionDeleteLabel(Request $request, Response $response): array
{
$this->checkAccess();
$body = $request->getParsedBody();
$id = $body->id ?? null;
if (!$id || !is_numeric($id)) {
throw new BadRequest('id (numeric) is required.');
}
return $this->getService()->deleteLabel((int) $id);
}
}
files/custom/Espo/Modules/KnowledgeBase/Resources/routes.json
+72
View File
@@ -126,5 +126,77 @@
"controller": "KnowledgeBase",
"action": "deleteTopic"
}
},
{
"route": "/KnowledgeBase/action/uploadBatch",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "uploadBatch"
}
},
{
"route": "/KnowledgeBase/action/batch",
"method": "get",
"params": {
"controller": "KnowledgeBase",
"action": "batch"
}
},
{
"route": "/KnowledgeBase/action/pendingBatches",
"method": "get",
"params": {
"controller": "KnowledgeBase",
"action": "pendingBatches"
}
},
{
"route": "/KnowledgeBase/action/commitJob",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "commitJob"
}
},
{
"route": "/KnowledgeBase/action/discardJob",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "discardJob"
}
},
{
"route": "/KnowledgeBase/action/labels",
"method": "get",
"params": {
"controller": "KnowledgeBase",
"action": "labels"
}
},
{
"route": "/KnowledgeBase/action/createLabel",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "createLabel"
}
},
{
"route": "/KnowledgeBase/action/mergeLabels",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "mergeLabels"
}
},
{
"route": "/KnowledgeBase/action/deleteLabel",
"method": "post",
"params": {
"controller": "KnowledgeBase",
"action": "deleteLabel"
}
}
]
files/custom/Espo/Modules/KnowledgeBase/Services/KnowledgeBaseService.php
+149
View File
@@ -227,6 +227,155 @@ class KnowledgeBaseService
return $this->request('DELETE', '/admin/kb/topics/' . $topicId, null);
}
// ── Phase 6 (v0.8.0): bulk upload + AI classifier + labels ──────────────
/**
* Multi-file upload. Each file is sent to /admin/kb/upload-batch in
* one multipart POST that repeats the `files` field. shira-hermes
* assigns the batch_id and fires per-file classify tasks.
*
* @param array<int,array{tmp_name:string,name:string,type:string,size:int}> $files
* @return array{batch_id:string, jobs:array<int,mixed>}
*/
public function uploadBatch(array $files, string $kind, int $topicId, string $username): array
{
if (!$files) {
throw new BadRequest('No files in batch.');
}
$url = $this->getBaseUrl() . '/admin/kb/upload-batch';
$apiKey = $this->getApiKey();
if (!$apiKey) {
throw new Error('SmartAssistant API key is not configured.');
}
$fields = [
'kind' => $kind,
'topic_id' => (string) $topicId,
];
// PHP cURL accepts repeated form fields by giving the array under
// a special "[]" suffix syntax — but actually CURLOPT_POSTFIELDS
// only sees the LAST value when the key repeats. Workaround:
// build the multipart body manually so we can repeat `files`.
$boundary = '----shira-hermes-' . bin2hex(random_bytes(8));
$body = '';
foreach ($fields as $k => $v) {
$body .= "--{$boundary}\r\n";
$body .= "Content-Disposition: form-data; name=\"{$k}\"\r\n\r\n";
$body .= $v . "\r\n";
}
foreach ($files as $f) {
$contents = @file_get_contents($f['tmp_name']);
if ($contents === false) {
throw new Error("Failed to read uploaded file: " . $f['name']);
}
$name = addslashes($f['name']);
$type = $f['type'] ?: $this->guessMime($f['name']);
$body .= "--{$boundary}\r\n";
$body .= "Content-Disposition: form-data; name=\"files\"; filename=\"{$name}\"\r\n";
$body .= "Content-Type: {$type}\r\n\r\n";
$body .= $contents . "\r\n";
}
$body .= "--{$boundary}--\r\n";
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $body,
CURLOPT_HTTPHEADER => [
'Accept: application/json',
'Content-Type: multipart/form-data; boundary=' . $boundary,
'X-Api-Key: ' . $apiKey,
'X-User-Name: ' . $username,
],
CURLOPT_RETURNTRANSFER => true,
// Batch upload of e.g. 10×10MB files is dominated by PHP→Python
// network transfer; bump from the single-file 120s.
CURLOPT_TIMEOUT => 300,
CURLOPT_CONNECTTIMEOUT => 10,
]);
$resp = curl_exec($ch);
$httpCode = (int) curl_getinfo($ch, CURLINFO_HTTP_CODE);
$err = curl_error($ch);
curl_close($ch);
if ($err) {
$this->log->error("KnowledgeBase: batch upload transport error: {$err}");
throw new Error("Failed to reach Knowledge Base: {$err}");
}
if ($httpCode === 413) {
throw new BadRequest('Total upload too large.');
}
if ($httpCode >= 400 && $httpCode < 500) {
$detail = $this->decodeDetail($resp) ?: 'Bad request';
throw new BadRequest($detail);
}
if ($httpCode < 200 || $httpCode >= 300) {
$this->log->error("KnowledgeBase: batch upload HTTP {$httpCode}: {$resp}");
throw new Error("Knowledge Base returned HTTP {$httpCode}");
}
$decoded = json_decode((string) $resp, true);
if (!is_array($decoded) || !isset($decoded['batch_id'])) {
throw new Error('Invalid response from Knowledge Base.');
}
return $decoded;
}
public function getBatch(string $batchId): array
{
return $this->get('/admin/kb/batch/' . rawurlencode($batchId));
}
public function listPendingBatches(int $limit): array
{
return $this->get('/admin/kb/batches?' . http_build_query(['limit' => $limit]));
}
public function commitJob(int $jobId, array $payload, string $username): array
{
return $this->postWithUser(
'/admin/kb/jobs/' . $jobId . '/commit',
$payload,
$username,
);
}
public function discardJob(int $jobId, string $username): array
{
return $this->postWithUser(
'/admin/kb/jobs/' . $jobId . '/discard',
null,
$username,
);
}
public function listLabels(?int $topicId, ?string $q, int $limit): array
{
$qs = ['limit' => $limit];
if ($topicId !== null) $qs['topic_id'] = $topicId;
if ($q !== null && $q !== '') $qs['q'] = $q;
return $this->get('/admin/kb/labels?' . http_build_query($qs));
}
public function createLabel(array $payload, string $username): array
{
return $this->postWithUser('/admin/kb/labels', $payload, $username);
}
public function mergeLabels(int $labelId, int $intoLabelId): array
{
return $this->request(
'POST',
'/admin/kb/labels/' . $labelId . '/merge',
['into_label_id' => $intoLabelId],
);
}
public function deleteLabel(int $labelId): array
{
return $this->request('DELETE', '/admin/kb/labels/' . $labelId, null);
}
private function postWithUser(string $path, ?array $payload, string $username): array
{
$url = $this->getBaseUrl() . $path;