granicus_archiver.legistar.guid_model

Data model using "Real" GUID's as identifiers

This may eventually replace the granicus_archiver.legistar.model module since its storage method is more robust. For now however, the two exist in parallel.

class granicus_archiver.legistar.guid_model.LegistarFile(name: KT, filename: Path, metadata: FileMeta, pdf_links_removed: bool = False)

Bases: LegistarFile

Information for a downloaded file within RGuidLegistarFiles.files

Parameters:

• name (KT)

• filename (Path)

• metadata (FileMeta)

• pdf_links_removed (bool)

property uid: LegistarFileUID

Unique id for the file

classmethod from_uid(uid: LegistarFileUID, filename: Path, metadata: FileMeta, pdf_links_removed: bool) → Self

Create an instance from a uid

Parameters:

• uid (LegistarFileUID)

• filename (Path)

• metadata (FileMeta)

• pdf_links_removed (bool)

Return type:

Self

class granicus_archiver.legistar.guid_model.AttachmentFile(name: KT, filename: Path, metadata: FileMeta, pdf_links_removed: bool = False)

Bases: AttachmentFile

Information for a downloaded attachment within RGuidLegistarFiles.files

Parameters:

• name (KT)

• filename (Path)

• metadata (FileMeta)

• pdf_links_removed (bool)

property uid: LegistarFileUID

Unique id for the file

classmethod from_uid(uid: LegistarFileUID, filename: Path, metadata: FileMeta, pdf_links_removed: bool) → Self

Create an instance from a uid

Parameters:

• uid (LegistarFileUID)

• filename (Path)

• metadata (FileMeta)

• pdf_links_removed (bool)

Return type:

Self

class granicus_archiver.legistar.guid_model.RGuidUpdateResult(changed: bool, link_keys: list[LegistarFileKey], attachment_keys: list[AttachmentName], files: dict[LegistarFileUID, LegistarFile | AttachmentFile], attributes: dict[str, Any])

Bases: NamedTuple

Parameters:

• changed (bool)

• link_keys (list[Literal['agenda', 'minutes', 'agenda_packet', 'video']])

• attachment_keys (list[AttachmentName])

• files (dict[LegistarFileUID, LegistarFile | AttachmentFile])

• attributes (dict[str, Any])

changed: bool

Whether any changes were made

link_keys: list[Literal['agenda', 'minutes', 'agenda_packet', 'video']]

Any URL attributes from DetailPageLinks that changed

attachment_keys: list[AttachmentName]

Any keys in DetailPageLinks.attachments that changed

files: dict[LegistarFileUID, LegistarFile | AttachmentFile]

Alias for field number 3

attributes: dict[str, Any]

Attributes of DetailPageResult that changed

class granicus_archiver.legistar.guid_model.RGuidLegistarFiles(base_dir: ~pathlib._local.Path, files: dict[~granicus_archiver.legistar.types.LegistarFileUID, ~granicus_archiver.legistar.guid_model.LegistarFile | ~granicus_archiver.legistar.guid_model.AttachmentFile] = <factory>)

Bases: Serializable

Collection of files within a RGuidDetailResult

Parameters:

• base_dir (Path)

• files (dict[LegistarFileUID, LegistarFile | AttachmentFile])

base_dir: Path

Base directory (relative to RGuidLegistarData.root_dir)

files: dict[LegistarFileUID, LegistarFile | AttachmentFile]

Mapping of LegistarFile or AttachmentFile objects using their uid as keys

property full_base_dir: Path

The complete file directory (base_dir prefixed with RGuidLegistarData.root_dir)

get_file_path(uid: LegistarFileUID, absolute: bool) → Path

Get the filename for the given uid

Parameters:

• uid (LegistarFileUID) – A LegistarFileUID

• absolute (bool) – If True the path will be within the full_base_dir, otherwise base_dir is used

Return type:

Path

add_file(uid: LegistarFileUID, meta: FileMeta, pdf_links_removed: bool) → LegistarFile | AttachmentFile

Add a file with the given uid

Parameters:

• uid (LegistarFileUID)

• meta (FileMeta)

• pdf_links_removed (bool)

Return type:

LegistarFile | AttachmentFile

ensure_local_hashes(check_existing: bool = False) → bool

Ensure that all local files have an sha1 hash stored in their metadata

Parameters:

check_existing (bool) – If True, the hash of the local file will be checked against the stored hash

Returns:

True if any hashes were generated or updated

Return type:

bool

class granicus_archiver.legistar.guid_model.RGuidDetailResult(page_url: URL, feed_guid: GUID, location: str, links: DetailPageLinks, agenda_status: AgendaStatus, minutes_status: MinutesStatus, feed_item: FeedItem, last_fake_stupid_guid: GUID | None = None)

Bases: DetailPageResult

Subclass of model.DetailPageResult for this module

Parameters:

• page_url (URL)

• feed_guid (GUID)

• location (str)

• links (DetailPageLinks)

• agenda_status (Literal['Final', 'Final-Addendum', 'Draft', 'Not Viewable by the Public'])

• minutes_status (Literal['Final', 'Final-Addendum', 'Draft', 'Not Viewable by the Public'])

• feed_item (FeedItem)

• last_fake_stupid_guid (GUID | None)

files: RGuidLegistarFiles

Instance of RGuidLegistarFiles

classmethod from_html(html_str: str | bytes, feed_item: FeedItem, parent: RGuidLegistarData) → Self

Create an instance from the raw html from page_url

Parameters:

• html_str (str | bytes)

• feed_item (FeedItem)

• parent (RGuidLegistarData)

Return type:

Self

property guid_compare: GuidCompare

A helper to compare GUID's

get_unique_folder() → Path

Get a local path to store files for this item

The folder structure will be:

<category>/<year>/<real_guid>

Where

<category>

Is the category of the feed_item

<year>

Is the 4-digit year of the meeting_date

<real_guid>

Is the real_guid

This makes it much less complex to ensure uniqueness compared to model.DetailPageResult.get_unique_folder(), but has a downside of being less user-friendly in a file browser (without the metadata).

Return type:

Path

copy() → Self

Create a deep copy of the instance

Return type:

Self

update(other: Self) → RGuidUpdateResult

Update self with changed attributes in other

Parameters:

other (Self)

Return type:

RGuidUpdateResult

class granicus_archiver.legistar.guid_model.RGuidLegistarData(root_dir: ~pathlib._local.Path, detail_results: dict[~granicus_archiver.legistar.types.REAL_GUID, ~granicus_archiver.legistar.guid_model.RGuidDetailResult] = <factory>, matched_guids: dict[~granicus_archiver.clips.model.CLIP_ID, ~granicus_archiver.legistar.types.REAL_GUID] = <factory>, items_by_clip_id: dict[~granicus_archiver.clips.model.CLIP_ID, ~granicus_archiver.legistar.guid_model.RGuidDetailResult] = <factory>, clip_id_overrides: dict[~granicus_archiver.legistar.types.REAL_GUID, ~granicus_archiver.clips.model.CLIP_ID | ~typing.Literal[_DoesNotExistEnum.DoesNotExist]] = <factory>)

Bases: AbstractLegistarModel[REAL_GUID, RGuidDetailResult]

Container for data gathered from Legistar, using real GUID's as keys

Parameters:

• root_dir (Path)

• detail_results (dict[REAL_GUID, RGuidDetailResult])

• matched_guids (dict[CLIP_ID, REAL_GUID])

• items_by_clip_id (dict[CLIP_ID, RGuidDetailResult])

• clip_id_overrides (dict[REAL_GUID, CLIP_ID | Literal[_DoesNotExistEnum.DoesNotExist]])

root_dir: Path

Root filesystem path for downloading assets

detail_results: dict[REAL_GUID, RGuidDetailResult]

Mapping of parsed RGuidDetailResult items with their feed_guid as keys

matched_guids: dict[CLIP_ID, REAL_GUID]

Clips that have been matched to FeedItems

items_by_clip_id: dict[CLIP_ID, RGuidDetailResult]

Mapping of items in detail_results with a valid clip_id

clip_id_overrides: dict[REAL_GUID, CLIP_ID | Literal[_DoesNotExistEnum.DoesNotExist]]

Mapping of items manually-linked to Clips

get_clip_id_for_guid(guid: REAL_GUID, use_overrides: bool = True) → CLIP_ID | None | Literal[_DoesNotExistEnum.DoesNotExist]

Get the clip id linked to the given guid

Parameters:

• guid (REAL_GUID) – The item feed_guid

• use_overrides (bool) – Whether to use items in clip_id_overrides (default is True)

Return type:

CLIP_ID | None | Literal[_DoesNotExistEnum.DoesNotExist]

Returns one of:

• clip_id (CLIP_ID)

  The matched Clip.id (if one was found)

• NoClip

  If the item has been explicitly set to have no Clip associated with it

• None

  If no match was found

create_item(html_str: str | bytes, feed_item: FeedItem, allow_update: Literal[True]) → tuple[RGuidDetailResult, RGuidUpdateResult | None]

create_item(html_str: str | bytes, feed_item: FeedItem, allow_update: Literal[False]) → RGuidDetailResult

Create and add an item from html

Parameters:

• html_str – The raw html to pass to RGuidDetailResult.from_html()

• feed_item – The rss_parser.FeedItem

• allow_update – If an item exists and this is True, its update() method will be called. Otherwise, a KeyError will be raised.

Returns:

• If allow_update is False this returns

  • item (RGuidDetailResult): The parsed item

• If allow_update is True this returns a tuple of

  • item (RGuidDetailResult): The parsed item

  • update_result: The RGuidUpdateResult if an item was updated (or None if no update was performed).

add_item(item: RGuidDetailResult) → None

Add an existing RGuidDetailResult object

Parameters:

item (RGuidDetailResult)

Return type:

None

add_detail_result(item: RGuidDetailResult) → None

Add a parsed RGuidDetailResult to detail_results

Parameters:

item (RGuidDetailResult)

Return type:

None

find_match_for_clip_id(clip_id: CLIP_ID) → RGuidDetailResult | None | Literal[_DoesNotExistEnum.DoesNotExist]

Find a RGuidDetailResult match for the given clip_id

Parameters:

clip_id (CLIP_ID)

Return type:

RGuidDetailResult | None | Literal[_DoesNotExistEnum.DoesNotExist]

is_clip_id_available(clip_id: CLIP_ID) → bool

Check whether the given clip id is linked to an item (returns True if there is no link)

Parameters:

clip_id (CLIP_ID)

Return type:

bool

is_guid_matched(guid: REAL_GUID) → bool

Check whether the item matching guid has a Clip associated with it

Parameters:

guid (REAL_GUID)

Return type:

bool

add_guid_match(clip_id: CLIP_ID, guid: REAL_GUID) → None

Add a Clip.id -> FeedItem match to matched_guids

This may seem redunant considering the find_match_for_clip_id() method, but is intended for adding matches for items without a video url to parse.

Parameters:

• clip_id (CLIP_ID)

• guid (REAL_GUID)

Return type:

None

add_clip_match_override(real_guid: REAL_GUID, clip_id: CLIP_ID | None | Literal[_DoesNotExistEnum.DoesNotExist]) → None

Add a manual override for the given real_guid

Parameters:

• real_guid (REAL_GUID) – The real_guid of the legistar item

• clip_id (CLIP_ID | None | Literal[_DoesNotExistEnum.DoesNotExist]) – The clip model.Clip.id matching the item. If NoClip is given, this signifies that the item should not have a Clip associated with it. If None is given, any previously added overrides for real_guid will be removed.

Return type:

None

iter_guid_matches() → Iterator[tuple[CLIP_ID, RGuidDetailResult]]

Iterate over items added by the add_guid_match(), add_guid_match() and add_clip_match_override() methods

Results are tuples of CLIP_ID and RGuidDetailResult

Return type:

Iterator[tuple[CLIP_ID, RGuidDetailResult]]

get_folder_for_item(item: REAL_GUID | RGuidDetailResult) → Path

Get a local path to store files for a RGuidDetailResult

This is the result of RGuidLegistarFiles.full_base_dir from RGuidDetailResult.files

Parameters:

item (REAL_GUID | RGuidDetailResult)

Return type:

Path

classmethod load(filename: PathLike, root_dir: Path | None = None) → Self

Loads an instance from previously saved data

Parameters:

• filename (PathLike)

• root_dir (Path | None)

Return type:

Self

save(filename: PathLike, indent: int | None = 2) → None

Saves all clip data as JSON to the given filename

Parameters:

• filename (PathLike)

• indent (int | None)

Return type:

None