cia-data-integration

The full skill.

Original instructions from the publisher’s SKILL.md

# CIA Data Integration Skill

## Purpose

Guide the integration of Swedish political data into the CIA platform's data model, covering entity mapping, data transformation, and consistency rules for Riksdagen, election, and financial data sources.

## When to Use

- ✅ Mapping new Riksdagen data types to JPA entities
- ✅ Processing parliamentary voting records
- ✅ Integrating election result data
- ✅ Adding new political entity relationships
- ✅ Maintaining data quality and consistency rules

Do NOT use for:
- ❌ Generic API integration (use api-integration skill)
- ❌ UI/dashboard development (use vaadin-component-design skill)

## CIA Data Model Overview

```
┌──────────────────────────────────────────────────────┐
│                   External Data Sources               │
├──────────────┬────────────┬──────────┬───────────────┤
│  Riksdagen   │  Val.se    │  World   │     ESV       │
│  Open Data   │  Elections │  Bank    │   Finances    │
└──────┬───────┴─────┬──────┴────┬─────┴───────┬───────┘
       │             │           │             │
       ▼             ▼           ▼             ▼
┌──────────────────────────────────────────────────────┐
│              Service Layer (Spring)                   │
│  service.external.riksdagen                          │
│  service.external.val                                │
│  service.external.worldbank                          │
│  service.external.esv                                │
└──────────────────────┬───────────────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────────────┐
│            Internal Data Model (JPA)                  │
│  model.internal.application.data                     │
│    ├── Politicians     ├── Committees                │
│    ├── Votes           ├── Documents                 │
│    ├── Parties         ├── Elections                 │
│    └── Risk Scores     └── Financial Data            │
└──────────────────────────────────────────────────────┘
```

## Key Module Structure

| Module | Purpose |
|--------|---------|
| `model.external.riksdagen` | API models for Riksdagen data |
| `model.external.riksdagen.person.impl` | Person/politician data mapping |
| `model.external.riksdagen.votering.impl` | Voting record processing |
| `model.external.riksdagen.dokumentstatus.impl` | Document metadata |
| `model.external.val.partier.impl` | Political party data |
| `model.internal.application.data.impl` | Internal aggregated model |

## Entity Mapping Patterns

### Riksdagen Person to Internal Politician

```java
@Entity
@Table(name = "person_data")
public class PersonData implements Serializable {

    @Id
    @Column(name = "id")
    private String id;              // Riksdagen person ID (intressent_id)

    @Column(name = "first_name")
    private String firstName;       // förnamn

    @Column(name = "last_name")
    private String lastName;        // efternamn

    @Column(name = "party")
    private String party;           // parti

    @Column(name = "born_year")
    private int bornYear;           // född_ar

    @Column(name = "gender")
    private String gender;          // kön

    @Column(name = "status")
    private String status;          // status (Tjänstgörande, etc.)
}
```

### Vote Record Processing

```java
@Entity
@Table(name = "vote_data")
public class VoteData implements Serializable {

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private Long id;

    @Column(name = "intressent_id")
    private String intressentId;    // Links to PersonData

    @Column(name = "rm")
    private String riksmote;        // Parliamentary session (e.g., "2023/24")

    @Column(name = "beteckning")
    private String designation;     // Vote designation

    @Column(name = "rost")
    private String vote;            // Ja/Nej/Avstår/Frånvarande

    @Column(name = "votering_id")
    private String voteringId;      // Unique voting session ID
}
```

## Data Quality Rules

### Validation Constraints

```java
public class DataQualityValidator {

    // Valid Swedish Parliament parties
    private static final Set<String> VALID_PARTIES = Set.of(
        "S", "M", "SD", "C", "V", "KD", "L", "MP", "-"
    );

    // Valid vote values from Riksdagen
    private static final Set<String> VALID_VOTES = Set.of(
        "Ja", "Nej", "Avstår", "Frånvarande"
    );

    public void validatePersonData(PersonData person) {
        Objects.requireNonNull(person.getId(), "Person ID is required");
        if (!VALID_PARTIES.contains(person.getParty())) {
            LOG.warn("Unknown party code: {} for person: {}",
                person.getParty(), person.getId());
        }
    }

    public void validateVoteData(VoteData vote) {
        Objects.requireNonNull(vote.getIntressentId(), "Intressent ID is required");
        if (!VALID_VOTES.contains(vote.getVote())) {
            throw new DataValidationException(
                "Invalid vote value: " + vote.getVote());
        }
    }
}
```

### Data Freshness Requirements

| Data Type | Maximum Age | Refresh Frequency |
|-----------|-------------|-------------------|
| Person list | 24 hours | Daily at 02:00 |
| Voting records | 1 hour | During Riksdag sessions |
| Documents | 6 hours | 4 times daily |
| Committee data | 24 hours | Daily |
| Election results | 30 days | Post-election only |

## Riksdagen API Integration

### API Endpoints

```
GET /personlista/?iid={id}&utformat=json     → Person details
GET /voteringlista/?rm={session}&utformat=json → Voting list
GET /dokumentlista/?sok={query}&utformat=json  → Document search
GET /dokumentstatus/{doc_id}.json              → Document status
```

### Data Import Workflow

```java
@Service
public class RiksdagDataImportService {

    @Transactional
    public ImportResult importPersonData() {
        // 1. Fetch from Riksdagen API
        PersonLista personList = riksdagClient.getPersonLista();

        // 2. Validate each record
        List<PersonData> validated = personList.getPersons().stream()
            .filter(this::isValid)
            .map(this::mapToInternal)
            .collect(Collectors.toList());

        // 3. Upsert into database
        int updated = personRepository.batchUpsert(validated);

        // 4. Log import metrics
        return new ImportResult(validated.size(), updated);
    }
}
```

## Security Considerations

- **Sanitize all imported data** — external data is untrusted input
- **Use parameterized queries** — never concatenate API data into SQL
- **Log data lineage** — track source and import timestamp for audit
- **GDPR compliance** — politician data is public, but handle personal data carefully
- **Validate XML/JSON** — protect against XXE and injection attacks

## ISMS Alignment

| Control | Requirement |
|---------|-------------|
| ISO 27001 A.8.11 | Data masking for sensitive fields |
| ISO 27001 A.5.33 | Protection of records |
| NIST CSF PR.DS-1 | Data-at-rest protection |
| CIS Control 3 | Data protection |