iDB (short for item database)

Description

  • Windows app + companion web interface
  • Drop, transfer, or pickup items
  • Virtual databases to separate out items
  • Item images (as of September 2019!)
  • Reseller support (as of October 2019!)
  • Auto‐refresh characters before they expire
  • Automated socketing (not Larzuk, but inserting gem/rune/jewel into bases) / cubing feature
  • Integration with WorldIsMine's AutoShop, Laz's ShopDrop and my SimpleShop
  • Integration with WooCommerce for order delivery
  • Integration with NeutronPlus & PermIt

Architecture

Hardware Requirements

Example hosts:

Kolbot Integration

Link

ShopDrop Integration

Link

Delivery Integration

Link

Codes

Codes are like products in a web shop (eg. Fortitude - Archon Plate - Ethereal - Bugged - 30% All Res, 1.5 HP/Lvl, 3552 Defense). They allow you to categorize your items and search for them easily. A Code consists of an ID (0 to 999,999), a description, a type (eg. Archon Plate) and a match/pickit (eg. Quality = Runeword & Name = Fortitude & Eth & Ebugged & AllResistances = 30 & LifePerLevel = 1.5 & Defense = 3552).

How to Write a Match/Pickit

Link

Special

There are 4 types of special codes. These allow you to create packs containing other codes! They can be multiple levels deep as well (A contains 1 B and 2 C's, and C contains 3 D's and 1 E). Special codes have a match format of [id]:[count],[id]:[count],.. (eg. 1:2,5:4,8:1).

Type Description
Special A plain pack
SpecialStrict A pack that fails immediately if only partially available, the app will do its best to make sure this pack is delivered in its entirety
SpecialAny Quantity on each ingredient is ignored, this pack resolves to the first code that is in stock (on the given realm/database)
SpecialAnyEven Quantity on each ingredient is ignored, this pack resolves to the code that has the highest stock (on the given realm/database)

Exporting

Drag your db.sqlite file onto dump-Codes.bat and it will export to a Codes-*.sql file.

  • This can be done even while iDB is running.
  • If you're about to do an import, it is always a good idea to back up your current set of codes.

Importing

Advanced users only!
Drag your *.sql file onto import-Codes.bat and it will import to db.sqlite.
After running iDB you will see a screen like this for a few minutes. If you have millions of items it can take up to an hour to reprocess all.


  • You should only do this if iDB is closed.
  • Since this operation does a full replacement, it must be full set of Codes, not just a partial set.
  • Be very careful not to change IDs of Codes that you are using with ShopDrop/AutoShop.
  • Be very careful not to mess up the formatting of your .sql file.

Images

Features

  • Item level is appended to the title
  • Unidentified uniques and sets will show the correct identified name (if possible):
  • Replenish over time stats (replenish quantity, etc) have the exact details:
  • Per level stats (life per level, etc) have the exact details:
  • Ethereal bugged items will have (Ebugged) appended to their title:
  • Upped items will have (Upped) appended to their title:

Hotkeys

While the image is showing, you can do CTRL+S to save the image to hard-drive.
While the image is showing, you can do CTRL+C to copy the image to clipboard.
While the image is showing, you can do CTRL+U to upload image to Imgur and copy link to clipboard.

Player Considerations

Be wary of items that potentially give +4 or more skills to non-owner classes, like Widowmaker which can spawn with up to +5 to Guided Arrow. In-game and in other kolbot-based droppers you might see that this Widowmaker has only +3 to Guided Arrow:
Due to the way oskills work, classes that own that skill (in this case Amazons) will be restricted to a maximum of +3. iDB will report the true value of +5 so you don't accidentally give away any godly items:
However, when calculating shield block and weapon speed, the item will be assumed to be held by the required class for the item, or if there is none, a sorceress. For level, strength and dexterity, all are assumed to be 1. Let me know if there is interest in having these choices configurable.

Databases

Databases are a virtual separation of items that allow the following configurations:
  • Priority for operations that span multiple databases (order preference, etc)
  • Whether or not web shop order delivery should be able to access
  • Whether or not ShopDrop order delivery should be able to access
  • Whether or not the characters should be auto-refreshed
  • Different limit IDs for using different keys & proxies via rDB
  • Special reseller access via iDB-Web

Processors

Databases are a virtual separation of items that allow the following configurations:

Name Description
Jobs Changes jobs from inactive to active (while obeying queue limits such as LimitPerRealm and also not activating a job for an account that is already active)
API: TCP Turns on a listener for PlainSocket network connections such as iDB-Web, PermIt, NeutronPlus, Kolbot, etc
API: HTTP Turns on a listener for WebSocket network connections such as iDB-Web, PermIt, NeutronPlus, Kolbot, etc
API: Allow New Drops If unchecked, drop requests will be responded with "Drops are temporarily disabled (service will restart soon)." (does not affect ShopDrop requests)
AutoRefresh Toggles the processor that checks for expiring characters and queues jobs to refresh them
Delivery Toggles the processor that communicates with the web shop for order delivery
ShopDrop Toggles the processor that communicates to the ShopDrop app (only controls iDB -> ShopDrop, to control ShopDrop -> iDB communication use the API: TCP processor)

Converting to non-ladder

When a ladder reset is coming close you should follow these steps.
  1. Uncheck all processors.
  2. Abort all jobs.
  3. Start a manual backup.
  4. A new set of files timestamped with the current date/time will be created. Wait a few minutes for the *-shm and *-wal files to disappear.
  5. In iDB's Items window, click on Convert to NL.

Recovery

Let's say you accidentally delete all your items and you're wondering how to restore them.
  1. Uncheck all processors.
  2. Abort all jobs.
  3. Close iDB.
  4. Open IIS and stop the website pool.
  5. Rename C:\Apps\iDB\db.sqlite to C:\Apps\iDB\db.sqlite.bak because it is a bad database now (or delete it).
  6. Rename latest backup file (like C:\Apps\iDB\db-2020-03-19-14.40.sqlite) to C:\Apps\iDB\db.sqlite.
  7. Open IIS and start website pool.
  8. Start iDB.

Important Notes

  1. Settings are stored in C:\Apps\iDB\iDB.xml.
  2. Databases, codes, items, etc are stored in C:\Apps\iDB\db.sqlite.
  3. Every 2 days your db.sqlite and your web logins are backed up to my server.

Changelog

2023-11-12: v1.0.0.735

  • ADD Added AllowJoiningPublicGames for trashers (default is true)

2023-03-26: v1.0.0.733

  • ADD Added AllowJoiningPublicGames for loggers (default is true)

2021-08-19: v1.0.0.731

  • ADD Added ShouldEnterGame for refreshers (default is false)

2021-07-23: v1.0.0.730

  • UPD Improved updating of character expirations

2021-07-20: v1.0.0.729

  • ADD Added button on Items window to delete expired characters (also respects realm/database filters)

2021-06-17: v1.0.0.726

  • ADD Added automatic job prioritization for Ladder

2021-06-12: v1.0.0.723

  • UPD Changed behavior to warn instead of deleting when encountering mismatched character core/ladder/expansion

2021-06-11: v1.0.0.722

  • ADD Added Move to Different Database button for currently filtered items

2021-06-08: v1.0.0.720

  • FIX Fixed a bug regarding reactivation of ShopDrop or AutoShop after a period of inactivity

2021-06-03: v1.0.0.719

  • UPD If wait-for-pickup is on, droppers will wait before dropping carry-only-1 items (annis, torches, gheeds, cube, etc) if there is already one on the ground

2021-05-29: v1.0.0.717

  • FIX Fixed bug with JSP shop processor not reporting failures for out of stock products
  • UPD Improved logging for JSP shop

2021-05-27: v1.0.0.715

  • FIX Minor fixes for builder

2021-05-26: v1.0.0.714

  • UPD Updater improvements
  • FIX Minor fixes

2021-05-18: v1.0.0.699

  • ADD Added support for mappings in SimpleShop

2021-05-17: v1.0.0.696

  • ADD Added new Deletion.log for account and character deletions
  • FIX Fixed expiration update bug (when sent from PermIt, etc)
  • UPD Updated expiration updates to delete missing characters (when sent from PermIt, etc)

2021-04-29: v1.0.0.689

  • ADD Added AutoRetryOrderOnGameFullBugged (default is true)

2021-04-29: v1.0.0.688

  • UPD Updated web drop messages

2021-04-22: v1.0.0.684

  • UPD Set DisableWest to default to false

2021-04-17: v1.0.0.681

  • ADD Added new option DisableWest (default is true) which disables drops from iDB-Web, Refreshers and Loggers for the West realm

2021-04-13: v1.0.0.680

  • UPD Adjusted abort job options to focus on refresher jobs

2021-04-12: v1.0.0.679

  • FIX Potential bug fix for errors on job ending

2021-04-10: v1.0.0.678

  • ADD Added builder request # to title

2021-04-09: v1.0.0.677

  • ADD Added ability to trash items (drop in public game)

2021-04-05: v1.0.0.674

  • ADD Added character details to summary for pickup jobs

2021-04-05: v1.0.0.673

  • ADD Added ability for loggers to attempt to join public games

2021-04-05: v1.0.0.672

  • ADD Added manual pickup game/password to module title

2021-03-25: v1.0.0.666

  • ADD Added WaitForPickupIfUnitPriceGreaterThan for SimpleShop

2021-03-23: v1.0.0.665

  • ADD Added Delete Items Matching Current Filters button on items tab

2021-03-18: v1.0.0.664

  • FIX Fixed decimal support for SimpleShop

2021-03-18: v1.0.0.663

  • ADD Added support for multiple SimpleShops

2021-03-07: v1.0.0.659

  • FIX Fixed decimal support for JSP Shop

2021-03-02: v1.0.0.658

  • ADD Added item image borders

2021-03-02: v1.0.0.657

  • UPD Performance improvement for searching items

2021-02-28: v1.0.0.656

  • ADD Added IP address to API logs

2021-02-28: v1.0.0.655

  • UPD Improved responsiveness of Items and Codes windows
  • UPD Builder now reports what ingredients were missing if it fails
  • UPD Pack editor sorts by ID
  • UPD Double-clicking in pack editor will add the code now

2021-02-23: v1.0.0.652

  • FIX Fixed using presets in Transfer mode

2021-02-23: v1.0.0.650

  • UPD Allow comments in SimpleShop definitions

2021-02-21: v1.0.0.649

  • FIX Fixed sorting by Name in Items window

2021-01-28: v1.0.0.642

  • ADD Added pickup/transfer presets

2021-01-28: v1.0.0.641

  • ADD Added LimitGlobal for modules
  • UPD LimitPerOrder and rDBIDDeliveryOverride are removed for certain modules

2021-01-25: v1.0.0.638

  • ADD Added SimpleShop

2021-01-24: v1.0.0.632

  • UPD Updated to infinite retries for sending order results

2020-12-29: v1.0.0.629

  • ADD Added handling for multiple server disconnects from lobby

2020-12-20: v1.0.0.628

  • FIX Quick fix for SpecialAny missing from db

2020-12-15: v1.0.0.626

  • UPD Massive log performance improvement

2020-12-15: v1.0.0.624

  • ADD Added JSPShop*-ConnectionTimeout for auto-disabling JSP shop updates if the app is having trouble connecting (for example, ShopDrop/AutoShop is not running but the iDB processor is left on)
  • UPD Stock updates are delayed and batched together to help with Log performance

2020-12-09: v1.0.0.619

  • UPD Re-indexed accounts & characters case-insensitively
  • UPD Upserting accounts & characters uses SQLite's ON CONFLICT feature for performance

2020-12-09: v1.0.0.618

  • ADD Added Get # of Items Matching Current Filters button on Items window
  • UPD Performance improvements during Log operations and more
  • UPD Removed prioritized db queue
  • ADD Added logging for Log operations to help track any slowness

2020-12-08: v1.0.0.615

  • ADD Added visualization of db vacuuming
  • UPD Prevented stacking of clean-and-reset operations

2020-12-07: v1.0.0.614

  • UPD Deleting a database works even when there are items attached now

2020-12-06: v1.0.0.613

  • UPD Improved jsp file import

2020-12-04: v1.0.0.611

  • FIX Fixed bug where backup got stuck

2020-12-04: v1.0.0.610

  • UPD Improved tracked item hash conversion to NL

2020-12-03: v1.0.0.608

  • ADD Added message that manual backup completed

2020-11-28: v1.0.0.607

  • FIX Fixed to limit Log operations

2020-11-28: v1.0.0.606

  • ADD Added limit to number of backgrounded Log db operations

2020-11-24: v1.0.0.605

  • ADD Added special code recursion detection

2020-11-11: v1.0.0.603

  • FIX Fixed counts for SpecialAny and SpecialAnyEven codes

2020-11-11: v1.0.0.602

  • UPD Improved game re-use by loggers

2020-11-09: v1.0.0.601

  • FIX Fixed bug with SpecialAnyEven codes

2020-11-06: v1.0.0.593

  • ADD Added date to tracked item hashes
  • ADD Added tracked price and date to Items window
  • UPD Items column state is persisted now

2020-11-04: v1.0.0.592

  • ADD Added support for Hash in pickits

2020-10-28: v1.0.0.590

  • ADD Added support for SpecialAnyEven codes

2020-10-13: v1.0.0.581

  • ADD Added prioritization system for db queue (Log operations are low priority)

2020-10-04: v1.0.0.575

  • ADD Added SystemOffline and DropFinished packets for ShopDrop

2020-10-04: v1.0.0.574

  • UPD Improved performance when jobs are ending

2020-09-30: v1.0.0.572

  • UPD Parallelized incoming API requests to improve responsiveness
  • FIX Fixed display order of jobs on main window

2020-08-12: v1.0.0.568

  • UPD Updated overhead messages to every 3 seconds

2020-07-28: v1.0.0.567

  • FIX Fixed picker switching to create after joining fails

2020-07-28: v1.0.0.566

  • ADD Added rDB ID override for manual pickup or transfer jobs

2020-07-12: v1.0.0.563

  • ADD Added PauseAndAbortAllOnRealmGoingDown

2020-06-28: v1.0.0.559

  • FIX Automatic detection and fixing of orders stuck in processing state

2020-06-27: v1.0.0.558

  • ADD Added LimitPerGameJspShop for JSP shop droppers

2020-06-15: v1.0.0.557

  • ADD Added rDBIDJspShopOverride for droppers

2020-06-15: v1.0.0.556

  • FIX Fixed broken job queue behavior

2020-06-11: v1.0.0.555

  • ADD Added database to the confirmed drops in JSP shop logs
  • ADD Added failed drops to JSP shop logs

2020-06-11: v1.0.0.552

  • UPD Made overhead pause message persistent

2020-06-09: v1.0.0.552

  • ADD Added overhead message for pause / resume

2020-06-08: v1.0.0.550

  • ADD Added ShopDrop-WaitForPickupMessage and AutoShop-WaitForPickupMessage that is spammed every 5 seconds

2020-06-07: v1.0.0.549

  • FIX Fixed queue performance degradation with large number of jobs

2020-06-06: v1.0.0.548

  • FIX Fixed bug where multi-tier strict special codes were not processing and aborting correctly

2020-06-06: v1.0.0.547

  • ADD Added pause and resume commands for picker

2020-06-05: v1.0.0.545

  • UPD Minor logging improvements

2020-06-03: v1.0.0.544

  • UPD Module terminates if lobby thread crashes

2020-06-03: v1.0.0.543

  • FIX Fixed bug where droppers didn't abort if builder never made it into a game

2020-06-02: v1.0.0.542

  • ADD Added account and character creation to builder
  • UPD Allow replacement builders to bypass queue restrictions
  • FIX Fixed droppers getting stuck waiting for a builder that aborted
  • FIX Added retries to cube transmuting
  • FIX Fixed a scenario where items were staying locked after an order

2020-05-25: v1.0.0.541

  • UPD Added checks to prevent recursive packs from being created

2020-05-25: v1.0.0.540

  • FIX Improved error handling if item fails to parse

2020-05-03: v1.0.0.539

  • FIX Fixed item graphics with spaces

2020-04-10: v1.0.0.537

  • FIX Fixed SystemOffline message for AutoShop

2020-04-10: v1.0.0.536

  • UPD Deleting items is more careful with cache clearing

2020-04-08: v1.0.0.535

  • ADD Added AutoShop support

2020-04-08: v1.0.0.533

  • FIX Fixed db import

2020-04-06: v1.0.0.529

  • FIX Fixed pickits with CrescentMoon name filter

2020-04-02: v1.0.0.527

  • FIX Fixed event handler memory leak and disposing of more SQLiteCommands

2020-04-02: v1.0.0.526

  • FIX Fixed pickits with *MaxDamage stat

2020-03-28: v1.0.0.523

  • FIX Fixed hotkey interference on some windows

2020-03-28: v1.0.0.522

  • FIX Fixed issue when importing ShopDrop data with different name or thumb

2020-03-24: v1.0.0.521

  • ADD Added pack editor for Special codes

2020-03-16: v1.0.0.519

  • ADD Added support for SpecialAny codes

2020-03-12: v1.0.0.518

  • ADD Added builder support for Special type

2020-02-28: v1.0.0.516

  • FIX Orders for hashes obey order-delivery-access flag
  • FIX Fixed db persistence of failed items on orders
  • UPD Synchronized job termination

2020-02-25: v1.0.0.514

  • FIX Logger now only tries to re-use game if game creation was successful

2020-02-25: v1.0.0.513

  • FIX Fixed caching of special codes even after deletion

2020-02-23: v1.0.0.511

  • FIX Fixed another graphic bug

2020-02-21: v1.0.0.510

  • FIX Fixed background backup/vacuum

2020-02-21: v1.0.0.508

  • UPD Backup/vacuuming happens in background now
  • UPD Updated SQLite to 1.0.112.0

2020-02-19: v1.0.0.507

  • FIX Added delay after trading to allow for replacement items to be received

2020-02-12: v1.0.0.506

  • UPD Updated ShopDrop window title

2020-02-09: v1.0.0.505

  • FIX Fixed droppers get stuck in infinite loop when code not found

2020-02-03: v1.0.0.503

  • UPD Builder updates
  • ADD Tcp support for gold, imbue, socket

2020-01-21: v1.0.0.502

  • FIX Fixed ebugged property

2020-01-19: v1.0.0.498

  • FIX Minor builder fixes

2020-01-15: v1.0.0.495

  • ADD Started logging gold/imbue/socket quest

2020-01-14: v1.0.0.493

  • FIX Fixed thumb for ShopDrops with graphic in pickit

2020-01-13: v1.0.0.492

  • ADD Added sync tool for tracked item hashes

2020-01-13: v1.0.0.488

  • ADD Added batching for tracked item hash publishing

2020-01-09: v1.0.0.485

  • ADD Added batch button for ShopDrop
  • FIX Fixed ordering of ShopDrop messages

2020-01-09: v1.0.0.483

  • UPD Removed Clear All Codes button

2020-01-08: v1.0.0.481

  • FIX Fixed failed codes friendly string for unsupported SKUs

2019-12-28: v1.0.0.476

  • UPD Logger should use same game if possible

2019-10-20: v1.0.0.405

  • ADD Droppers will terminate other droppers if game does not exist

2019-09-29: v1.0.0.390

  • ADD Added item images

2017-02-08: v1.0.0.0

  • ADD Initial