Is a bulk upload of attributes possible?

[jloehel]

We handle some huge csv files with attributes. The upload method through the UI is not really stable for huge files. It times out or crashes if the file has more than ~300,000 entries. It would be nice to see there something async.

Instead I wrote a few upload tools with the help of pymisp. It's better but still improvable. Uploading/updating the attributes one by one is error-prone and really slow. I fixed it with retries but it bloats now the log files. ^^ It would be nice to upload something like X attributes in a single request. Is it possible via pymisp/MISP?

... or is it better to let MISP pull the attributes with local csv feeds and provide scripts to upload the CSV files?

[github-germ]

Re' using PyMISP to upload large attributed events... we find it generally best to build the MISPEvent in memory in the client, and then upload iterating through batches (e.g. 1k attributes within that MISPEvent per batch).

• If this is a new event start with pymisp.add_event followed by pymisp.update_event for the remaining batches.
• For an existing event in MISP, populate the MISPEvent by calling pymisp.search_index(...) followed by pymisp.update_event for the remaining batches.

[jloehel]

Thanks, that's much faster. 👍