Building a Context-Aware Security Assessment Engine

IT teams managing Microsoft 365 are responsible for Conditional Access policies, device compliance rules, app registrations, SharePoint settings, and more. For teams managing multiple tenants, keeping track of what is configured correctly across all of them is overwhelming.

TL;DR: We are building Lora Ops, a security assessment engine that evaluates Microsoft 365 tenants using live Graph API data, scores them based on context (not just binary pass/fail), and generates phased remediation plans. It is designed for IT teams who need to manage security at scale without treating every tenant the same.

The two approaches that do not work

Most IT teams rely on one of two methods today:

1. Manual checklists - an engineer logs into each tenant, checks settings one by one, writes a report. Takes hours per tenant. Does not scale.
2. Third-party scanning tools - they identify issues but lack context. A 5-person startup and a 5,000-person enterprise get the same generic recommendations.

Neither approach understands context. A missing Conditional Access policy is critical for an enterprise but might be acceptable for a micro-tenant that only uses basic Microsoft 365. Security is not one-size-fits-all, but most tools treat it that way.

The assessment cycle

Lora Ops follows a cycle that IT teams already understand:

Assess - Run security checks against a tenant's actual configuration via the Microsoft Graph API. Not a static scan. Live data, evaluated with an understanding of what "enforced" actually means versus "exists but not doing anything yet."

Plan - Generate a phased remediation plan with specific actions, each linked to the API endpoint needed to fix it. Actions include risk levels, expected outcomes, and rollback steps. The team reviews and approves before anything changes.

Execute - The team carries out the approved plan. Today this is manual. The system tracks progress, not because we cannot automate it, but because automated security changes carry real liability risk. Trust has to be earned before we put a button that modifies production Conditional Access policies.

Verify - Re-run the assessment. Compare before and after. See which actions moved the needle and which need further work.

Then repeat. Security is a cycle, not a project.

Scoring that understands context

Most security tools score on a binary: is MFA enabled or not? We score on enforcement and coverage.

A Conditional Access policy requiring MFA exists but is in report-only mode? That is not a pass. MFA is required for admins but not all users? That is partial coverage, not a full pass.

The scoring also adjusts based on tenant size. A small company gets heavier weight on identity protection (their biggest risk surface). A larger company gets heavier weight on device compliance (more endpoints to manage). This means IT teams can compare scores across tenants fairly. A score of 75 represents a similar level of security maturity regardless of tenant size.

We are still tuning these weights based on real-world data. It is not perfect yet, but it is meaningfully better than treating every tenant identically.

Grounded in real data

Every assessment check is backed by a knowledge base of Graph API documentation that includes the exact API calls, what the responses look like from real tenants (not just Microsoft's docs), what the fields mean, and edge cases discovered from real-world usage.

When the AI evaluates a check, it does not guess. It has the response shape, the field definitions, and notes from production experience. This is the difference between "the AI thinks MFA is configured" and "the AI verified the actual policy state and enforcement level."

Checks that improve over time

Microsoft changes API responses. New properties appear. Old ones get deprecated. A check that worked last month might return unexpected data today.

The assessment engine detects these issues automatically. When it encounters an unexpected response or a data quality problem, it records what went wrong and proposes a fix. An admin reviews and applies it. The check definition improves, and the next assessment run works better.

Over time, the system gets more accurate without anyone writing code. The checks evolve based on what is actually happening in production tenants.

Plain-language check creation

The newest piece is semantic search integration. Instead of requiring deep Graph API knowledge to create new assessment checks, an admin describes what they want to check in plain language.

"Check if guest accounts are inactive for 90 days." The system searches the knowledge base by meaning, finds relevant API endpoints, checks if similar tests already exist, and generates a complete check definition. The admin reviews, tests against a real tenant, and approves.

This is how we plan to scale from dozens of checks to hundreds. The barrier to creating new checks dropped from "know the Graph API inside out" to "describe a security concern."

What is next

Custom assessment profiles - IT teams will be able to select which checks matter for their organization. Healthcare compliance needs different checks than financial services.

Integrated remediation - Connect approved plan actions to actual policy changes through the Graph API. Click "execute" on an action, the system applies the change with proper rollback. This ships when the trust is proven through manual execution first.

Continuous monitoring - Scheduled assessments that detect drift. A tenant's score drops? Your team gets alerted with exactly which check started failing and why.

The honest picture

We are not pretending this is finished. Lora Ops is in active development with early partners. The assessment cycle works, scoring is live, and check creation through semantic search is functional. Integrated remediation and continuous monitoring are next.

The market is moving from "scan and report" to "assess, plan, execute, and maintain." We think the key differentiator is context awareness, not just finding problems but understanding which problems matter for each specific tenant. That is what we are building toward.

Frequently Asked Questions

Does Lora Ops modify tenant configurations automatically?

Not today. The system generates remediation plans and tracks progress, but an engineer executes the changes manually. Automated execution is on the roadmap, but only after we have built enough trust through manual workflows.

How is this different from Microsoft Secure Score?

Secure Score gives you Microsoft's general recommendations. Lora Ops evaluates against context-aware checks that adjust for tenant size and industry, generates phased remediation plans with rollback steps, and improves its checks over time based on real-world data.

Can teams create their own checks?

Yes. Describe a security concern in plain language and the system generates a check definition using semantic search across the knowledge base. The admin reviews and tests before it goes live.

What Microsoft 365 services does it cover?

Currently: Conditional Access, Intune device configuration, Entra ID settings, and SharePoint. Coverage expands as we add more checks to the knowledge base.